Hi Linus,
I think some of your physical connections are wrong.
However, rather than troubleshoot based on your description, I think a few suggestions might help you more.
1. Do you have the datasheet for the micro in your language? Go to the Atmel website, they offer the datasheet for free. You really need it.
2. There are many guides on the internet for how to "blink and LED" with an AVR micro. You should consult one that offers photos and shows the circuit connection. I have the Atmega 32 and Attiny84 parts sharing a breadboard and I have been following the YouTube videos of a particular website. This one is pretty good, perhaps you can get it translated to your preferred language. I'm ashamed to say that all I understand is English.
https://www.youtube.com/results?search_query=newbiehack%20microcontroller&sm=123. If you have a personal website somewhere, you might want to take photos of your project under good lighting and post the photos there. You want to make sure that circuit connections are easy to separate and identify in the photos. Then perhaps someone might be able to view the photos and help you. I don't think list members normally send photos to the avr chat list. They likely get removed anyhow by the mailing list software.