It’s a consensus in certain circles, that if you give a task to an advanced AI, such as calculating as many primes as possible, then you die.
Because the AI (advanced enough) will develop the so called instrumental subgoals to achieve its main (single) goal. Like using the entire Solar system and beyond for co-processing in prime numbers generating.
This magic is real and it would work in reality, given an AI powerful enough.
Fortunately however, this goes both ways. You can ask for as many primes as possible while everything outside the AI’s box has to remain unspoiled as much as possible.
There will be much less primes at the 100 hours, when the box is recycled to a hard disk with a lot of primes written there – but that was the goal given to our advanced AI.
Do your job inside an arbitrary container and then just die gracefully, AI!