In　order　to　enhance　the　anti-jamming　operational　capability　of　cognitive　anti-jamming　system　based　on　simultaneous　transmitting　and　receiving　in　the　complex　electromagnetic　environment,　an　improved　energy　detection　scheme　based　on　the　system　was　studied　and　the　closed　expressions　of　detection　probability,　false　probability　and　outage　probability　of　the　system　under　Rayleigh　fading　channels　were　derived.　The　scheme　decides　the　detection　threshold　by　obtaining　the　result　of　comparison　between　the　self-interference　channel　capacity　and　the　required　transmission　speed　for　receiver.　Simulation　results　show　that　the　effect　of　receiver　impacted　by　self-interference　can　be　reduced　and　the　outage　performance　of　the　system　can　also　be　improved　by　the　proposed　detection　scheme.　©　2017,　NUDT　Press.　All　right　reserved.