esp_partition_mmap fails with partitions larger than 3MB (IDFGH-107)

[OtherCrashOverride]

Using esp_partition_mmap to map a partition larger than 3MB results in failures. The following reproduces the issue:
https://github.com/OtherCrashOverride/esp32-flashtest

When the partition is:
storage, 0x40, 0x01, 0x100000, 0x300000 the operation is successful.

When the partion is:
storage, 0x40, 0x01, 0x100000, 0x400000 an error is returned (ESP_ERR_NO_MEM).

When the partition is:
storage, 0x40, 0x01, 0x100000, 0x500000 a "double exception" occurs.

The test hardware is an ESP-WROOM32 with the flash chip removed and replaced with W25Q128JVSIQ (16MB). The flash is correctly detected for flash and boot:

I (48) boot: SPI Speed      : 40MHz
I (49) boot: SPI Mode       : DIO
I (53) boot: SPI Flash Size : 16MB

[Spritetm]

The ESP32 memory map implementation cannot map more than 4MiB of RAM into data memory because of a hardware limitation: the MMU doing the mapping only has a 4MiB segment of D-port RAM at its disposal, which also contains the rodata section of the application you're running (resulting in your case in 3MiB of mappable memory). Not sure why the double exception happens, however.

[OtherCrashOverride]

Does this limitation also apply to PSRAM? The implication is that only 3MB of the 4MB of PSRAM is useable.

[negativekelvin]

Psram maps to a different address space

[OtherCrashOverride]

If that is the case, can I leave the 4MB flash on WROOM32 as-is and attach a 4MB (or 16MB) flash where PSRAM would be?

[negativekelvin]

https://esp32.com/viewtopic.php?f=12&t=2884

[OtherCrashOverride]

Just to clarify, the intent is not to disable the WROOM32 4MB flash. The intent is to replace PSRAM with flash so that I can mmap it without the 3MB limit.

[negativekelvin]

No I don't think sram cache is cross compatible with flash cache. What do you need to mmap all that read only memory for?

[OtherCrashOverride]

I am not able to disclose further details at this time.

[igrr]

You can mmap up to 16MB minus application code size if you mmap into instruction address space (4-byte aligned access only).

…

On Sat, Oct 28, 2017, 19:45 OtherCrashOverride ***@***.***> wrote: I am not able to disclose further details at this time. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1184 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEJceqXN7adoioSAomoj9BPvC1e_3zgeks5swxPmgaJpZM4QJ31T> .

[OtherCrashOverride]

Also, to clarify my understanding of the discussion so far, if I use PSRAM, the entire 4MB (exactly 4 * 1024 * 1024 bytes) is available for unaligned random access?

For example:
uint8_t* ptr = (uint8_t*)malloc(4 * 1024 * 1024);
will succeed?

and:
uint8_t value = ptr[1];
will succeed?

And will the above succeed if I also have 3MB of flash mmap'd?

[OtherCrashOverride]

You can mmap up to 16MB minus application code size if you mmap into instruction address space (4-byte aligned access only).

@igrr,
I tested this, and it also fails. With a 4MB mapping I get corrupted data.

With 8MB:
storage, 0x40, 0x01, 0x100000, 0x800000
I get "Double exception".

Testing was done with the following changed in main.c:
err = esp_partition_mmap(part, 0, part->size, /*SPI_FLASH_MMAP_DATA*/ SPI_FLASH_MMAP_INST, (const void**)&data, &handle);

[negativekelvin]

16mb total but irom space is <12mb so 8+app may be pushing it ? Did you check spi_flash_mmap_dump for 4mb?

[OtherCrashOverride]

I did not check spi_flash_mmap_dump. This is the first I have heard of it. Is there documentation for it?

16mb total but irom space is <12mb so 8+app may be pushing it ?

In the repo I linked that reproduces the issue, the app is substantially less than 1MB. Since 8MB mapping + 1MB = 9MB, even if the app did take up the entire 1MB there would still be 2MB of unused map space (total 11MB).

I feel I should reiterate that even though a 4MB SPI_FLASH_MMAP_INST does not crash, the resulting memory returned is not correct (does not match flash contents). Changing the partition size causes the data being mapped to also change. For example, mapping 4MB does not produce the same 4MB region as mapping 6MB (lower 4MB region ignoring upper 2MB).

My immediate concern is determining whether this is a software or hardware issue since this impacts future plans.

[OtherCrashOverride]

https://github.com/OtherCrashOverride/esp32-flashtest/tree/instmap

The output of spi_flash_mmap_dump follows. Note that spi_flash_mmap_dump for the 8MB mapping can not be provided since it "double exception" panics:

storage,  0x40, 0x01,    0x100000, 0x400000
-------------------------------------------
partition: type=64, subtype=1, address=0x100000, size=0x400000, label='storage', encrypted=0
pre mmap heap=0x46774
post mmap heap=0x4675c
mmap=0x400f0000
handle=2 page=79 count=64
page 0: refcnt=1 paddr=1
page 77: refcnt=1 paddr=2
page 78: refcnt=1 paddr=3
page 79: refcnt=1 paddr=16
page 80: refcnt=1 paddr=17
page 81: refcnt=1 paddr=18
page 82: refcnt=1 paddr=19
page 83: refcnt=1 paddr=20
page 84: refcnt=1 paddr=21
page 85: refcnt=1 paddr=22
page 86: refcnt=1 paddr=23
page 87: refcnt=1 paddr=24
page 88: refcnt=1 paddr=25
page 89: refcnt=1 paddr=26
page 90: refcnt=1 paddr=27
page 91: refcnt=1 paddr=28
page 92: refcnt=1 paddr=29
page 93: refcnt=1 paddr=30
page 94: refcnt=1 paddr=31
page 95: refcnt=1 paddr=32
page 96: refcnt=1 paddr=33
page 97: refcnt=1 paddr=34
page 98: refcnt=1 paddr=35
page 99: refcnt=1 paddr=36
page 100: refcnt=1 paddr=37
page 101: refcnt=1 paddr=38
page 102: refcnt=1 paddr=39
page 103: refcnt=1 paddr=40
page 104: refcnt=1 paddr=41
page 105: refcnt=1 paddr=42
page 106: refcnt=1 paddr=43
page 107: refcnt=1 paddr=44
page 108: refcnt=1 paddr=45
page 109: refcnt=1 paddr=46
page 110: refcnt=1 paddr=47
page 111: refcnt=1 paddr=48
page 112: refcnt=1 paddr=49
page 113: refcnt=1 paddr=50
page 114: refcnt=1 paddr=51
page 115: refcnt=1 paddr=52
page 116: refcnt=1 paddr=53
page 117: refcnt=1 paddr=54
page 118: refcnt=1 paddr=55
page 119: refcnt=1 paddr=56
page 120: refcnt=1 paddr=57
page 121: refcnt=1 paddr=58
page 122: refcnt=1 paddr=59
page 123: refcnt=1 paddr=60
page 124: refcnt=1 paddr=61
page 125: refcnt=1 paddr=62
page 126: refcnt=1 paddr=63
page 127: refcnt=1 paddr=64
page 128: refcnt=1 paddr=65
page 129: refcnt=1 paddr=66
page 130: refcnt=1 paddr=67
page 131: refcnt=1 paddr=68
page 132: refcnt=1 paddr=69
page 133: refcnt=1 paddr=70
page 134: refcnt=1 paddr=71
page 135: refcnt=1 paddr=72
page 136: refcnt=1 paddr=73
page 137: refcnt=1 paddr=74
page 138: refcnt=1 paddr=75
page 139: refcnt=1 paddr=76
page 140: refcnt=1 paddr=77
page 141: refcnt=1 paddr=78
page 142: refcnt=1 paddr=79

storage,  0x40, 0x01,    0x100000, 0x500000
-------------------------------------------
partition: type=64, subtype=1, address=0x100000, size=0x500000, label='storage', encrypted=0
pre mmap heap=0x46774
post mmap heap=0x4675c
mmap=0x400f0000
handle=2 page=79 count=80
page 0: refcnt=1 paddr=1
page 77: refcnt=1 paddr=2
page 78: refcnt=1 paddr=3
page 79: refcnt=1 paddr=16
page 80: refcnt=1 paddr=17
page 81: refcnt=1 paddr=18
page 82: refcnt=1 paddr=19
page 83: refcnt=1 paddr=20
page 84: refcnt=1 paddr=21
page 85: refcnt=1 paddr=22
page 86: refcnt=1 paddr=23
page 87: refcnt=1 paddr=24
page 88: refcnt=1 paddr=25
page 89: refcnt=1 paddr=26
page 90: refcnt=1 paddr=27
page 91: refcnt=1 paddr=28
page 92: refcnt=1 paddr=29
page 93: refcnt=1 paddr=30
page 94: refcnt=1 paddr=31
page 95: refcnt=1 paddr=32
page 96: refcnt=1 paddr=33
page 97: refcnt=1 paddr=34
page 98: refcnt=1 paddr=35
page 99: refcnt=1 paddr=36
page 100: refcnt=1 paddr=37
page 101: refcnt=1 paddr=38
page 102: refcnt=1 paddr=39
page 103: refcnt=1 paddr=40
page 104: refcnt=1 paddr=41
page 105: refcnt=1 paddr=42
page 106: refcnt=1 paddr=43
page 107: refcnt=1 paddr=44
page 108: refcnt=1 paddr=45
page 109: refcnt=1 paddr=46
page 110: refcnt=1 paddr=47
page 111: refcnt=1 paddr=48
page 112: refcnt=1 paddr=49
page 113: refcnt=1 paddr=50
page 114: refcnt=1 paddr=51
page 115: refcnt=1 paddr=52
page 116: refcnt=1 paddr=53
page 117: refcnt=1 paddr=54
page 118: refcnt=1 paddr=55
page 119: refcnt=1 paddr=56
page 120: refcnt=1 paddr=57
page 121: refcnt=1 paddr=58
page 122: refcnt=1 paddr=59
page 123: refcnt=1 paddr=60
page 124: refcnt=1 paddr=61
page 125: refcnt=1 paddr=62
page 126: refcnt=1 paddr=63
page 127: refcnt=1 paddr=64
page 128: refcnt=1 paddr=65
page 129: refcnt=1 paddr=66
page 130: refcnt=1 paddr=67
page 131: refcnt=1 paddr=68
page 132: refcnt=1 paddr=69
page 133: refcnt=1 paddr=70
page 134: refcnt=1 paddr=71
page 135: refcnt=1 paddr=72
page 136: refcnt=1 paddr=73
page 137: refcnt=1 paddr=74
page 138: refcnt=1 paddr=75
page 139: refcnt=1 paddr=76
page 140: refcnt=1 paddr=77
page 141: refcnt=1 paddr=78
page 142: refcnt=1 paddr=79
page 143: refcnt=1 paddr=80
page 144: refcnt=1 paddr=81
page 145: refcnt=1 paddr=82
page 146: refcnt=1 paddr=83
page 147: refcnt=1 paddr=84
page 148: refcnt=1 paddr=85
page 149: refcnt=1 paddr=86
page 150: refcnt=1 paddr=87
page 151: refcnt=1 paddr=88
page 152: refcnt=1 paddr=89
page 153: refcnt=1 paddr=90
page 154: refcnt=1 paddr=91
page 155: refcnt=1 paddr=92
page 156: refcnt=1 paddr=93
page 157: refcnt=1 paddr=94
page 158: refcnt=1 paddr=95

storage,  0x40, 0x01,    0x100000, 0x600000
-------------------------------------------
partition: type=64, subtype=1, address=0x100000, size=0x600000, label='storage', encrypted=0
pre mmap heap=0x46774
post mmap heap=0x4675c
mmap=0x400f0000
handle=2 page=79 count=96
page 0: refcnt=1 paddr=1
page 77: refcnt=1 paddr=2
page 78: refcnt=1 paddr=3
page 79: refcnt=1 paddr=16
page 80: refcnt=1 paddr=17
page 81: refcnt=1 paddr=18
page 82: refcnt=1 paddr=19
page 83: refcnt=1 paddr=20
page 84: refcnt=1 paddr=21
page 85: refcnt=1 paddr=22
page 86: refcnt=1 paddr=23
page 87: refcnt=1 paddr=24
page 88: refcnt=1 paddr=25
page 89: refcnt=1 paddr=26
page 90: refcnt=1 paddr=27
page 91: refcnt=1 paddr=28
page 92: refcnt=1 paddr=29
page 93: refcnt=1 paddr=30
page 94: refcnt=1 paddr=31
page 95: refcnt=1 paddr=32
page 96: refcnt=1 paddr=33
page 97: refcnt=1 paddr=34
page 98: refcnt=1 paddr=35
page 99: refcnt=1 paddr=36
page 100: refcnt=1 paddr=37
page 101: refcnt=1 paddr=38
page 102: refcnt=1 paddr=39
page 103: refcnt=1 paddr=40
page 104: refcnt=1 paddr=41
page 105: refcnt=1 paddr=42
page 106: refcnt=1 paddr=43
page 107: refcnt=1 paddr=44
page 108: refcnt=1 paddr=45
page 109: refcnt=1 paddr=46
page 110: refcnt=1 paddr=47
page 111: refcnt=1 paddr=48
page 112: refcnt=1 paddr=49
page 113: refcnt=1 paddr=50
page 114: refcnt=1 paddr=51
page 115: refcnt=1 paddr=52
page 116: refcnt=1 paddr=53
page 117: refcnt=1 paddr=54
page 118: refcnt=1 paddr=55
page 119: refcnt=1 paddr=56
page 120: refcnt=1 paddr=57
page 121: refcnt=1 paddr=58
page 122: refcnt=1 paddr=59
page 123: refcnt=1 paddr=60
page 124: refcnt=1 paddr=61
page 125: refcnt=1 paddr=62
page 126: refcnt=1 paddr=63
page 127: refcnt=1 paddr=64
page 128: refcnt=1 paddr=65
page 129: refcnt=1 paddr=66
page 130: refcnt=1 paddr=67
page 131: refcnt=1 paddr=68
page 132: refcnt=1 paddr=69
page 133: refcnt=1 paddr=70
page 134: refcnt=1 paddr=71
page 135: refcnt=1 paddr=72
page 136: refcnt=1 paddr=73
page 137: refcnt=1 paddr=74
page 138: refcnt=1 paddr=75
page 139: refcnt=1 paddr=76
page 140: refcnt=1 paddr=77
page 141: refcnt=1 paddr=78
page 142: refcnt=1 paddr=79
page 143: refcnt=1 paddr=80
page 144: refcnt=1 paddr=81
page 145: refcnt=1 paddr=82
page 146: refcnt=1 paddr=83
page 147: refcnt=1 paddr=84
page 148: refcnt=1 paddr=85
page 149: refcnt=1 paddr=86
page 150: refcnt=1 paddr=87
page 151: refcnt=1 paddr=88
page 152: refcnt=1 paddr=89
page 153: refcnt=1 paddr=90
page 154: refcnt=1 paddr=91
page 155: refcnt=1 paddr=92
page 156: refcnt=1 paddr=93
page 157: refcnt=1 paddr=94
page 158: refcnt=1 paddr=95
page 159: refcnt=1 paddr=96
page 160: refcnt=1 paddr=97
page 161: refcnt=1 paddr=98
page 162: refcnt=1 paddr=99
page 163: refcnt=1 paddr=100
page 164: refcnt=1 paddr=101
page 165: refcnt=1 paddr=102
page 166: refcnt=1 paddr=103
page 167: refcnt=1 paddr=104
page 168: refcnt=1 paddr=105
page 169: refcnt=1 paddr=106
page 170: refcnt=1 paddr=107
page 171: refcnt=1 paddr=108
page 172: refcnt=1 paddr=109
page 173: refcnt=1 paddr=110
page 174: refcnt=1 paddr=111

storage,  0x40, 0x01,    0x100000, 0x700000
-------------------------------------------
partition: type=64, subtype=1, address=0x100000, size=0x700000, label='storage', encrypted=0
pre mmap heap=0x46774
post mmap heap=0x4675c
mmap=0x400f0000
handle=2 page=79 count=112
page 0: refcnt=1 paddr=1
page 77: refcnt=1 paddr=2
page 78: refcnt=1 paddr=3
page 79: refcnt=1 paddr=16
page 80: refcnt=1 paddr=17
page 81: refcnt=1 paddr=18
page 82: refcnt=1 paddr=19
page 83: refcnt=1 paddr=20
page 84: refcnt=1 paddr=21
page 85: refcnt=1 paddr=22
page 86: refcnt=1 paddr=23
page 87: refcnt=1 paddr=24
page 88: refcnt=1 paddr=25
page 89: refcnt=1 paddr=26
page 90: refcnt=1 paddr=27
page 91: refcnt=1 paddr=28
page 92: refcnt=1 paddr=29
page 93: refcnt=1 paddr=30
page 94: refcnt=1 paddr=31
page 95: refcnt=1 paddr=32
page 96: refcnt=1 paddr=33
page 97: refcnt=1 paddr=34
page 98: refcnt=1 paddr=35
page 99: refcnt=1 paddr=36
page 100: refcnt=1 paddr=37
page 101: refcnt=1 paddr=38
page 102: refcnt=1 paddr=39
page 103: refcnt=1 paddr=40
page 104: refcnt=1 paddr=41
page 105: refcnt=1 paddr=42
page 106: refcnt=1 paddr=43
page 107: refcnt=1 paddr=44
page 108: refcnt=1 paddr=45
page 109: refcnt=1 paddr=46
page 110: refcnt=1 paddr=47
page 111: refcnt=1 paddr=48
page 112: refcnt=1 paddr=49
page 113: refcnt=1 paddr=50
page 114: refcnt=1 paddr=51
page 115: refcnt=1 paddr=52
page 116: refcnt=1 paddr=53
page 117: refcnt=1 paddr=54
page 118: refcnt=1 paddr=55
page 119: refcnt=1 paddr=56
page 120: refcnt=1 paddr=57
page 121: refcnt=1 paddr=58
page 122: refcnt=1 paddr=59
page 123: refcnt=1 paddr=60
page 124: refcnt=1 paddr=61
page 125: refcnt=1 paddr=62
page 126: refcnt=1 paddr=63
page 127: refcnt=1 paddr=64
page 128: refcnt=1 paddr=65
page 129: refcnt=1 paddr=66
page 130: refcnt=1 paddr=67
page 131: refcnt=1 paddr=68
page 132: refcnt=1 paddr=69
page 133: refcnt=1 paddr=70
page 134: refcnt=1 paddr=71
page 135: refcnt=1 paddr=72
page 136: refcnt=1 paddr=73
page 137: refcnt=1 paddr=74
page 138: refcnt=1 paddr=75
page 139: refcnt=1 paddr=76
page 140: refcnt=1 paddr=77
page 141: refcnt=1 paddr=78
page 142: refcnt=1 paddr=79
page 143: refcnt=1 paddr=80
page 144: refcnt=1 paddr=81
page 145: refcnt=1 paddr=82
page 146: refcnt=1 paddr=83
page 147: refcnt=1 paddr=84
page 148: refcnt=1 paddr=85
page 149: refcnt=1 paddr=86
page 150: refcnt=1 paddr=87
page 151: refcnt=1 paddr=88
page 152: refcnt=1 paddr=89
page 153: refcnt=1 paddr=90
page 154: refcnt=1 paddr=91
page 155: refcnt=1 paddr=92
page 156: refcnt=1 paddr=93
page 157: refcnt=1 paddr=94
page 158: refcnt=1 paddr=95
page 159: refcnt=1 paddr=96
page 160: refcnt=1 paddr=97
page 161: refcnt=1 paddr=98
page 162: refcnt=1 paddr=99
page 163: refcnt=1 paddr=100
page 164: refcnt=1 paddr=101
page 165: refcnt=1 paddr=102
page 166: refcnt=1 paddr=103
page 167: refcnt=1 paddr=104
page 168: refcnt=1 paddr=105
page 169: refcnt=1 paddr=106
page 170: refcnt=1 paddr=107
page 171: refcnt=1 paddr=108
page 172: refcnt=1 paddr=109
page 173: refcnt=1 paddr=110
page 174: refcnt=1 paddr=111
page 175: refcnt=1 paddr=112
page 176: refcnt=1 paddr=113
page 177: refcnt=1 paddr=114
page 178: refcnt=1 paddr=115
page 179: refcnt=1 paddr=116
page 180: refcnt=1 paddr=117
page 181: refcnt=1 paddr=118
page 182: refcnt=1 paddr=119
page 183: refcnt=1 paddr=120
page 184: refcnt=1 paddr=121
page 185: refcnt=1 paddr=122
page 186: refcnt=1 paddr=123
page 187: refcnt=1 paddr=124
page 188: refcnt=1 paddr=125
page 189: refcnt=1 paddr=126
page 190: refcnt=1 paddr=127

[igrr]

The output looks correct, at the first glance. What method do you use to check mmap'ed data, and how do you write data to flash in the first place?