I was looking through the lwip and I found some obvious problems in the way invalidate_cpu_cache() is used.
There are two factors which the usage of invalidate_cpu_cache() fails to consider:
1. invalidate_cpu_cache() invalidates cache lines, and therefore works on cache line boundaries. For example, if the cache line size is 32 bytes and invalidate_cpu_cache() is called on a buffer starting at 0x1008, this buffer starts 8 bytes into a 32-byte cache line, and the first eight bytes before the buffer (0x1000 to 0x1007) will also be invalidated. The same problem exists at the end of the buffer, e.g. data after the buffer will be invalidated if the buffer address and size are not exact multiples of the cache line size.
2. From what I can tell, the memory alignment used for memory allocation in lwIP is 1, because opt.h defines MEM_ALIGNMENT as 1.
So basically, when invalidate_cpu_cache() is called on a buffer, data may be inadvertently lost because the data in buffers before and after the buffer to invalidate is also invalidated.
Toshi