Re: [lwip-users] Why does enabling Debug fix this tcp

Rather than flushing cache you can create a non-cached section of RAM in the linker script and put buffers there.

.ram_nocache (NOLOAD):
{
. = ALIGN(4);
_s_ram_nocache = .;
*(.ram_nocache)
. = ALIGN(4);
_e_ram_nocache = .;
} > ram_nocache

.ram_nocache_data : AT (_etext + SIZEOF(.relocate))
{
. = ALIGN(4);
_s_ram_nocache_vma = .;
_s_ram_nocache_lma = LOADADDR(.ram_nocache_data);
*(.ram_nocache_data)
. = ALIGN(4);
_e_ram_nocache_vma = .;
} > ram_nocache

Then in your start up code you configure the MPU to disable the cache for this area of code:

// START THE NO CACHE DATA SEGMENT
/* Initialize the no cache data segment */
pSrc = &_s_ram_nocache_lma;
pDest = &_s_ram_nocache_vma;
if (pSrc != pDest) {
for (; pDest < &_e_ram_nocache_vma;) {
*pDest++ = *pSrc++;
}
}

/* Clear the no cache zero segment */
for (pDest = &_s_ram_nocache; pDest < &_e_ram_nocache;) {
*pDest++ = 0;
}
// END THE NO CACHE DATA SEGMENT

#define SRAM_NOCACHE_START_ADDRESS (uint32_t)(&_s_ram_nocache)
#define NOCACHE_SRAM_REGION_SIZE ((uint32_t)()&_e_ram_nocache) - (uint32_t)(&_s_ram_nocache))
#define MPU_NOCACHE_SRAM_REGION (11)
#define INNER_OUTER_NORMAL_NOCACHE_TYPE(x) ((0x01 << MPU_RASR_TEX_Pos ) | ( 0 <<MPU_RASR_C_Pos ) | \
( 0 << MPU_RASR_B_Pos ) | ( x << MPU_RASR_S_Pos))
/* Disable the MPU region */
MPU->CTRL = 0; //MPU_CTRL_ENABLE_Msk;

uint32_t dw_region_base_addr = SRAM_NOCACHE_START_ADDRESS |
MPU_RBAR_VALID_Msk |
MPU_NOCACHE_SRAM_REGION;
uint32_t dw_region_attr = 0x03<< MPU_RASR_AP_Pos|
INNER_OUTER_NORMAL_NOCACHE_TYPE( 1 ) |
(15<<1) | //set size to 2^16 as we are at end of SRAM and memory above not used
//mpu_cal_mpu_region_size(NOCACHE_SRAM_REGION_SIZE) |
MPU_RASR_ENABLE_Msk;
MPU->RBAR = dw_region_base_addr;
MPU->RASR = dw_region_attr;
/* Enable the MPU region */
MPU->CTRL = (MPU_CTRL_ENABLE_Msk | MPU_CTRL_PRIVDEFENA_Msk);
__DSB();
__ISB();

Then you can put your buffers in this section of RAM and it will not be cached.

__attribute__ ((section (".ram_nocache"), aligned (8))) uint8_t mydata[...];

For the ATSAM part the GMAC driver used a RAM buffer and internal DMA. So the driver checked a byte in the RAM buffer to see when the DMA was done, basically polled that byte. Since the buffer was cached the byte never changed as the core had no idea the DMA was happening.

This same problem occurred anytime DMA was used, be it using DMA peripheral or with other peripherals (USB and GMAC) that had internal DMA engines.

One option is to invalidate cache before reading these buffers, the other is to use a non-cached region of RAM for the buffers.

Like you I questioned the timing, so I put buffers in the non-cached region and verified that this did fix the problem. Now I was confident that this was indeed the problem and not a timing problem.

Trampas

On Thu, Mar 25, 2021 at 4:06 AM Osborne, David <David.Osborne@leicabiosystems.com> wrote:

Damn you L1 cache! I would disable you permanently if you weren’t so damn fast. :)

It appears cleaning the cache over the tx buffers address space has fixed the issue. My only concern is, it shouldn’t have. The tx buffers are set up as Normal Write through memory (T=0, C=1, B=0, S=0). I have a sneaky suspicion cleaning the cache is simply providing the delay that the debug was (but hey a fix is a fix)

@Alister Fisher I am using your Ethernet patches. I was hoping to switch back to the mainstream by now, but I’m not confident ST has fixed all the issues yet.

Thanks for your help everyone.

Cheers

David

From: lwip-users <lwip-users-bounces+david.osborne=leicabiosystems.com@nongnu.org> On Behalf Of Trampas Stern
Sent: Thursday, 25 March 2021 4:10 AM
To: Mailing list for lwIP users <lwip-users@nongnu.org>
Subject: Re: [lwip-users] Why does enabling Debug fix this tcp_sndbuf()

I had similar problem with an ATSAME70 which was an M7 core. Turns out that the ethernet driver from Microchip did not invalidate cache for the ethernet MAC driver. Hence it would work some of the time and not others, like when debugging was on. Disabling the data cache on the processor would allow code to run with no errors.

Microchip told me that none of their drivers were designed to work with cache on and to run ANY sample code or drivers required instruction and data cache to be turned off. Needless to say I rewrote their drivers.

Also note the need for barrier instructions (__DSB(), __ISB()) these are often needed and vendor's drivers are missing them. For example barriers are often needed at end of interrupt handlers. Note even then some peripherals have caches in the peripherals that can not be flushed with barrier instructions, for example Flash memory.

I also ran into this as that right after writing to flash memory I would read back the wrong values due to flash peripheral caching. I had to create my own cache flush by reading large block of flash no near where I wrote, then returning to where I wrote and reading again. The peripheral driver had a cache flush that did not work, so the reading different region was the only way to ensure the flash cache was flushed.

The function below disables the cache in the core, part of core_cm7.h

/**
\brief Disable D-Cache
\details Turns off D-Cache
*/
__STATIC_INLINE void SCB_DisableDCache (void)
{
#if defined (__DCACHE_PRESENT) && (__DCACHE_PRESENT == 1U)
register uint32_t ccsidr;
register uint32_t sets;
register uint32_t ways;

SCB->CSSELR = 0U; /*(0U << 1U) | 0U;*/ /* Level 1 data cache */
__DSB();

SCB->CCR &= ~(uint32_t)SCB_CCR_DC_Msk; /* disable D-Cache */
__DSB();

ccsidr = SCB->CCSIDR;

/* clean & invalidate D-Cache */
sets = (uint32_t)(CCSIDR_SETS(ccsidr));
do {
ways = (uint32_t)(CCSIDR_WAYS(ccsidr));
do {
SCB->DCCISW = (((sets << SCB_DCCISW_SET_Pos) & SCB_DCCISW_SET_Msk) |
((ways << SCB_DCCISW_WAY_Pos) & SCB_DCCISW_WAY_Msk) );
#if defined ( __CC_ARM )
__schedule_barrier();
#endif
} while (ways-- != 0U);
} while(sets-- != 0U);

__DSB();
__ISB();
#endif
}

On Wed, Mar 24, 2021 at 3:11 PM goldsimon@gmx.de <goldsimon@gmx.de> wrote:

Am 24.03.2021 um 16:32 schrieb Osborne, David:
> Hi All,
>
>
>
> I’m using LwIP + FreeRTOS + Sockets on STM32H7 with external SDRAM and
> FLASH (Ethernet buffers are internal). It works well until the server
> has to send a very large message (> 18Kbytes). One in five attempts
> ends in failure. During the transmission tcp_sndbuf() (in api_msg.c)
> returns zero and the connection falls over. From what I’ve read, a
> common cause is that it’s waiting for an outstanding ACK. I’m no
> expert, but it looks like the ACK has been sent (see Wireshark attachment).
>
>
>
> When I enable certain debug settings it works every time (don’t you hate
> that).

That's an indication that there's something wrong that's hidden by
either longer runtime (more debug code enabled) or by heavier cache
usage (more code/data in the caches).

In any case, I'd suggest thoroughly review the OS port and the netif
driver regarding caching issues and/or threading issues (where interrupt
vs. thread/main loop also counts in).

Regards,
Simon

_______________________________________________
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.
_______________________________________________
lwip-users mailing list
lwip-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users

From:	Trampas Stern
Subject:	Re: [lwip-users] Why does enabling Debug fix this tcp_sndbuf()
Date:	Thu, 25 Mar 2021 07:44:06 -0400