Messages in this thread | | | Date | Wed, 10 Feb 2010 13:28:19 +0000 | Subject | PROBLEM: heavy network traffic triggers system crash | From | Kovalev Igor <> |
| |
1. Summary: heavy network traffic triggers "recvmsg bug" and system crash
2. Have being running Ubuntu (9.04 to 9.10) on Toshiba Satellite L40-139 laptop. Heavy network traffic systematically triggers a crash (either kernel panic or segmentation fault). Among applications that have triggered system crash are: firefox (all versions since 3.2.3); system update manager, a Java-based application downloading via http/ftp. Usually the problem appears while downloading at speed of 150kBytes/s or more within 20-30 min. Just before the moment of crash, tons of messages like: Feb 3 02:30:04 kovalev-home kernel: [35474.511460] recvmsg bug: copied BFA5DAF9 seq BFA5E0A1 fill up system logs (preceeded with application warning). The problem has been observed first with kernel 2.6.28-13 and persists through 2.6.31-19, all have been installed from official Ubuntu repositories. Network connection is via Atheros AR5007eg wireless (reported as AR5001 by ath5k), with several different routers and providers. Since Windows Vista had never crashed by similar network traffic on this machine, this doesn't look like a hardware fault.
Examples of warning right before crash (syslog): --------------- Feb 2 12:51:38 kovalev-home kernel: [76478.454177] WARNING: at /build/buildd/linux-2.6.31/net/ipv4/tcp.c:1408 tcp_recvmsg+0xa49/0xb20() Feb 2 12:51:38 kovalev-home kernel: [76478.454179] Hardware name: Satellite L40 Feb 2 12:51:38 kovalev-home kernel: [76478.454181] Modules linked in: sbp2 xt_tcpudp aes_i586 aes_generic binfmt_misc ppdev nls_iso8859_1 nls_cp437 vfat fat joydev snd_hda_codec_analog arc4 ecb pcmcia snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy iptable_filter snd_seq_oss ip_tables x_tables snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse serio_raw yenta_socket rsrc_nonstatic pcmcia_core snd shpchp ath5k mac80211 ath cfg80211 soundcore snd_page_alloc asus_laptop led_class video1394 raw1394 lp parport usbhid fbcon tileblit font bitblit softcursor 8139too 8139cp mii ohci1394 ieee1394 i915 drm i2c_algo_bit intel_agp agpgart video output Feb 2 12:51:38 kovalev-home kernel: [76478.454230] Pid: 2357, comm: transmission Tainted: G W 2.6.31-18-generic #55-Ubuntu Feb 2 12:51:38 kovalev-home kernel: [76478.454233] Call Trace: Feb 2 12:51:38 kovalev-home kernel: [76478.454237] [<c014513d>] warn_slowpath_common+0x6d/0xa0 Feb 2 12:51:38 kovalev-home kernel: [76478.454242] [<c04d2439>] ? tcp_recvmsg+0xa49/0xb20 Feb 2 12:51:38 kovalev-home kernel: [76478.454245] [<c04d2439>] ? tcp_recvmsg+0xa49/0xb20 Feb 2 12:51:38 kovalev-home kernel: [76478.454250] [<c0145185>] warn_slowpath_null+0x15/0x20 Feb 2 12:51:38 kovalev-home kernel: [76478.454254] [<c04d2439>] tcp_recvmsg+0xa49/0xb20 Feb 2 12:51:38 kovalev-home kernel: [76478.454259] [<c0492433>] sock_common_recvmsg+0x43/0x60 Feb 2 12:51:38 kovalev-home kernel: [76478.454263] [<c04920ac>] sock_aio_read+0x11c/0x130 Feb 2 12:51:38 kovalev-home kernel: [76478.454267] [<c01e78fc>] do_sync_read+0xbc/0x100 Feb 2 12:51:38 kovalev-home kernel: [76478.454272] [<c015c1d0>] ? autoremove_wake_function+0x0/0x40 Feb 2 12:51:38 kovalev-home kernel: [76478.454276] [<c04f02c1>] ? inet_ioctl+0x31/0xb0 Feb 2 12:51:38 kovalev-home kernel: [76478.454280] [<c02cbb7f>] ? security_file_permission+0xf/0x20 Feb 2 12:51:38 kovalev-home kernel: [76478.454284] [<c01e799f>] ? rw_verify_area+0x5f/0xe0 Feb 2 12:51:38 kovalev-home kernel: [76478.454287] [<c0490940>] ? sock_ioctl+0x0/0x260 Feb 2 12:51:38 kovalev-home kernel: [76478.454291] [<c01e7fda>] vfs_read+0x17a/0x190 Feb 2 12:51:38 kovalev-home kernel: [76478.454295] [<c01f5651>] ? do_vfs_ioctl+0x71/0x310 Feb 2 12:51:38 kovalev-home kernel: [76478.454299] [<c0217e0d>] ? sys_epoll_ctl+0x9d/0x240 Feb 2 12:51:38 kovalev-home kernel: [76478.454303] [<c01e850d>] sys_read+0x3d/0x70 Feb 2 12:51:38 kovalev-home kernel: [76478.454307] [<c01033ac>] syscall_call+0x7/0xb Feb 2 12:51:38 kovalev-home kernel: [76478.454309] ---[ end trace 1d3e832a4fd8afb9 ]--- Feb 2 12:51:38 kovalev-home kernel: [76478.454312] recvmsg bug: copied 7B0ADBE3 seq 7B0AE18B --------------- Feb 3 02:52:43 kovalev-home kernel: [36833.898061] WARNING: at /build/buildd/linux-2.6.31/net/ipv4/tcp.c:1470 tcp_recvmsg+0x9b6/0xb20() Feb 3 02:52:43 kovalev-home kernel: [36833.898065] Hardware name: Satellite L40 Feb 3 02:52:43 kovalev-home kernel: [36833.898067] Modules linked in: xt_tcpudp aes_i586 aes_generic binfmt_misc ppdev nls_iso8859_1 nls_cp437 vfat fat joydev snd_hda_codec_analog arc4 ecb pcmcia snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss iptable_filter ip_tables x_tables snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device psmouse serio_raw yenta_socket rsrc_nonstatic pcmcia_core shpchp ath5k mac80211 ath cfg80211 snd soundcore snd_page_alloc asus_laptop led_class video1394 raw1394 lp parport usbhid fbcon tileblit font bitblit softcursor ohci1394 ieee1394 8139too 8139cp mii i915 drm i2c_algo_bit intel_agp agpgart video output Feb 3 02:52:43 kovalev-home kernel: [36833.898126] Pid: 22266, comm: java Tainted: G W 2.6.31-18-generic #55-Ubuntu Feb 3 02:52:43 kovalev-home kernel: [36833.898129] Call Trace: Feb 3 02:52:43 kovalev-home kernel: [36833.898136] [<c014513d>] warn_slowpath_common+0x6d/0xa0 Feb 3 02:52:43 kovalev-home kernel: [36833.898141] [<c04d23a6>] ? tcp_recvmsg+0x9b6/0xb20 Feb 3 02:52:43 kovalev-home kernel: [36833.898145] [<c04d23a6>] ? tcp_recvmsg+0x9b6/0xb20 Feb 3 02:52:43 kovalev-home kernel: [36833.898150] [<c0145185>] warn_slowpath_null+0x15/0x20 Feb 3 02:52:43 kovalev-home kernel: [36833.898154] [<c04d23a6>] tcp_recvmsg+0x9b6/0xb20 Feb 3 02:52:43 kovalev-home kernel: [36833.898159] [<c0492433>] sock_common_recvmsg+0x43/0x60 Feb 3 02:52:43 kovalev-home kernel: [36833.898163] [<c04908c0>] sock_recvmsg+0xf0/0x120 Feb 3 02:52:43 kovalev-home kernel: [36833.898167] [<c01b346d>] ? generic_file_buffered_write+0x5d/0x150 Feb 3 02:52:43 kovalev-home kernel: [36833.898173] [<c015c1d0>] ? autoremove_wake_function+0x0/0x40 Feb 3 02:52:43 kovalev-home kernel: [36833.898177] [<c01b4e13>] ? __generic_file_aio_write_nolock+0x1f3/0x510 Feb 3 02:52:43 kovalev-home kernel: [36833.898191] [<f8e466e5>] ? ath5k_tx_processq+0x2a5/0x340 [ath5k] Feb 3 02:52:43 kovalev-home kernel: [36833.898196] [<c01338bd>] ? __enqueue_entity+0x9d/0xc0 Feb 3 02:52:43 kovalev-home kernel: [36833.898200] [<c0490e33>] sys_recvfrom+0xb3/0x130 Feb 3 02:52:43 kovalev-home kernel: [36833.898205] [<c015fcdc>] ? ktime_get_ts+0x4c/0x60 Feb 3 02:52:43 kovalev-home kernel: [36833.898209] [<c015fd06>] ? ktime_get+0x16/0x40 Feb 3 02:52:43 kovalev-home kernel: [36833.898213] [<c0573f61>] ? _spin_unlock_bh+0x11/0x20 Feb 3 02:52:43 kovalev-home kernel: [36833.898217] [<c0492c04>] ? release_sock+0x94/0xa0 Feb 3 02:52:43 kovalev-home kernel: [36833.898221] [<c04d1642>] ? tcp_ioctl+0x72/0x170 Feb 3 02:52:43 kovalev-home kernel: [36833.898225] [<c04d15d0>] ? tcp_ioctl+0x0/0x170 Feb 3 02:52:43 kovalev-home kernel: [36833.898230] [<c0512881>] ? inet6_ioctl+0x31/0xd0 Feb 3 02:52:43 kovalev-home kernel: [36833.898233] [<c04909a8>] ? sock_ioctl+0x68/0x260 Feb 3 02:52:43 kovalev-home kernel: [36833.898236] [<c0490ee6>] sys_recv+0x36/0x40 Feb 3 02:52:43 kovalev-home kernel: [36833.898240] [<c0491cf0>] sys_socketcall+0x190/0x270 Feb 3 02:52:43 kovalev-home kernel: [36833.898245] [<c0165341>] ? do_gettimeofday+0x11/0x40 Feb 3 02:52:43 kovalev-home kernel: [36833.898249] [<c01033ac>] syscall_call+0x7/0xb Feb 3 02:52:43 kovalev-home kernel: [36833.898252] ---[ end trace d1a4a3665ef673d0 ]--- --------
3. SYSTEM CONFIGURATION
Ubuntu 9.10; Linux kernel 2.6.31-18 (currently 2.6.31-19.56) and Gnome 2.28.1
Networking (running via wlan0): eth0 Link encap:Ethernet HWaddr 00:1a:92:fc:ad:97 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) Interrupt:16 Base address:0xd800
lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:183847 errors:0 dropped:0 overruns:0 frame:0 TX packets:183847 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:46933290 (46.9 MB) TX bytes:46933290 (46.9 MB)
wlan0 Link encap:Ethernet HWaddr 00:1b:9e:4a:2e:f8 inet addr:192.168.1.65 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::21b:9eff:fe4a:2ef8/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:5432610 errors:0 dropped:0 overruns:0 frame:0 TX packets:5681225 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:156835732 (156.8 MB) TX bytes:60892985 (60.8 MB)
wmaster0 Link encap:UNSPEC HWaddr 00-1B-9E-4A-2E-F8-34-61-00-00-00-00-00-00-00-00 UP RUNNING MTU:0 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
iwconfig output: wlan0 IEEE 802.11bg ESSID:"BTHomeHub2-HKZ2" Mode:Managed Frequency:2.457 GHz Access Point: 00:24:17:C7:72:3B Bit Rate=54 Mb/s Tx-Power=20 dBm Retry long limit:7 RTS thr:off Fragment thr:off Encryption key:CC35-06B9-0F61-D80D-01EA-582C-2887-81AA-B663-D88C-8A79-B717-E020-7AEF-C85F-5A06 [2] Power Management:off Link Quality=58/70 Signal level=-52 dBm Noise level=-102 dBm Rx invalid nwid:0 Rx invalid crypt:0 Rx invalid frag:0 Tx excessive retries:0 Invalid misc:0 Missed beacon:0
7.2. /proc/cpuinfo: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 22 model name : Intel(R) Celeron(R) M CPU 520 @ 1.60GHz stepping : 1 cpu MHz : 1595.985 cache size : 1024 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss tm pbe nx lm constant_tsc up arch_perfmon pebs bts pni dtes64 monitor ds_cpl tm2 ssse3 cx16 xtpr pdcm lahf_lm bogomips : 3191.97 clflush size : 64 power management:
7.3. /proc/modules udf 80900 0 - Live 0xf8a93000 crc_itu_t 1852 1 udf, Live 0xf8165000 isofs 31620 0 - Live 0xf85a5000 sbp2 22888 1 - Live 0xf94d1000 aes_i586 8124 1 - Live 0xf85f7000 aes_generic 27484 1 aes_i586, Live 0xf85b0000 binfmt_misc 8356 1 - Live 0xf817c000 ppdev 6688 0 - Live 0xf8094000 nls_iso8859_1 3740 1 - Live 0xf8226000 nls_cp437 5372 1 - Live 0xf8208000 vfat 10716 1 - Live 0xf8174000 fat 51452 1 vfat, Live 0xf8582000 snd_hda_codec_analog 59292 1 - Live 0xf89ea000 arc4 1660 2 - Live 0xf8a3f000 ecb 2524 2 - Live 0xf8a36000 snd_hda_intel 26920 2 - Live 0xf8a26000 snd_hda_codec 75708 2 snd_hda_codec_analog,snd_hda_intel, Live 0xf8a04000 snd_hwdep 7200 1 snd_hda_codec, Live 0xf8591000 snd_pcm_oss 37920 0 - Live 0xf84ea000 snd_mixer_oss 16028 1 snd_pcm_oss, Live 0xf8278000 video1394 15452 0 - Live 0xf8272000 snd_pcm 75296 3 snd_hda_intel,snd_hda_codec,snd_pcm_oss, Live 0xf85e2000 snd_seq_dummy 2656 0 - Live 0xf816e000 snd_seq_oss 28576 0 - Live 0xf8215000 snd_seq_midi 6432 0 - Live 0xf80f8000 joydev 10240 0 - Live 0xf8159000 snd_rawmidi 22208 1 snd_seq_midi, Live 0xf8147000 snd_seq_midi_event 6940 2 snd_seq_oss,snd_seq_midi, Live 0xf80b4000 raw1394 25308 0 - Live 0xf827e000 snd_seq 50224 6 snd_seq_dummy,snd_seq_oss,snd_seq_midi,snd_seq_midi_event, Live 0xf8228000 snd_timer 22276 2 snd_pcm,snd_seq, Live 0xf820d000 snd_seq_device 6920 5 snd_seq_dummy,snd_seq_oss,snd_seq_midi,snd_rawmidi,snd_seq, Live 0xf80fb000 iptable_filter 3100 0 - Live 0xf8091000 snd 59204 16 snd_hda_codec_analog,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_pcm_oss,snd_mixer_oss,snd_pcm,snd_seq_oss,snd_rawmidi,snd_seq,snd_timer,snd_seq_device, Live 0xf8287000 ip_tables 11692 1 iptable_filter, Live 0xf8142000 pcmcia 36808 0 - Live 0xf809e000 ath5k 124260 0 - Live 0xf80bf000 soundcore 7264 1 snd, Live 0xf808d000 psmouse 56500 0 - Live 0xf8e53000 mac80211 181140 1 ath5k, Live 0xf8e0c000 ath 8060 1 ath5k, Live 0xf8dce000 snd_page_alloc 9156 2 snd_hda_intel,snd_pcm, Live 0xf8dc1000 cfg80211 93052 3 ath5k,mac80211,ath, Live 0xf8da2000 yenta_socket 24296 1 - Live 0xf8d64000 rsrc_nonstatic 11644 1 yenta_socket, Live 0xf8d51000 pcmcia_core 36528 3 pcmcia,yenta_socket,rsrc_nonstatic, Live 0xf8d3c000 asus_laptop 17400 0 - Live 0xf8d24000 shpchp 32272 0 - Live 0xf8d0e000 led_class 4096 2 ath5k,asus_laptop, Live 0xf8cfa000 lp 8964 0 - Live 0xf8cee000 x_tables 16544 1 ip_tables, Live 0xf8cde000 serio_raw 5280 0 - Live 0xf8cbf000 parport 35340 2 ppdev,lp, Live 0xf8cac000 usbhid 38208 0 - Live 0xf8595000 fbcon 36640 72 - Live 0xf84f6000 tileblit 2460 1 fbcon, Live 0xf84e7000 font 8124 1 fbcon, Live 0xf84df000 bitblit 5372 1 fbcon, Live 0xf8178000 softcursor 1756 1 bitblit, Live 0xf8171000 i915 221320 3 - Live 0xf82a4000 drm 159584 3 i915, Live 0xf8236000 i2c_algo_bit 5760 1 i915, Live 0xf815e000 ohci1394 29900 2 video1394, Live 0xf814f000 ieee1394 86596 4 sbp2,video1394,raw1394,ohci1394, Live 0xf80e0000 8139too 22620 0 - Live 0xf80b7000 8139cp 19260 0 - Live 0xf8097000 mii 5212 2 8139too,8139cp, Live 0xf8039000 intel_agp 27484 2 i915, Live 0xf80ab000 agpgart 34988 2 drm,intel_agp, Live 0xf8044000 video 19380 1 i915, Live 0xf803d000 output 2780 1 video, Live 0xf8020000
7.4. /proc/ioports 0000-001f : dma1 0020-0021 : pic1 0040-0043 : timer0 0050-0053 : timer1 0060-0060 : keyboard 0064-0064 : keyboard 0070-0071 : rtc0 0080-008f : dma page reg 00a0-00a1 : pic2 00c0-00df : dma2 00f0-00ff : fpu 0170-0177 : 0000:00:1f.2 0170-0177 : ata_piix 01f0-01f7 : 0000:00:1f.2 01f0-01f7 : ata_piix 0250-025f : pnp 00:08 0250-025f : pnp 00:0a 0376-0376 : 0000:00:1f.2 0376-0376 : ata_piix 03c0-03df : vga+ 03f6-03f6 : 0000:00:1f.2 03f6-03f6 : ata_piix 0400-041f : pnp 00:08 0400-041f : pnp 00:0a 0480-04bf : 0000:00:1f.0 0480-04bf : pnp 00:08 04d0-04d1 : pnp 00:08 0800-087f : 0000:00:1f.0 0800-087f : pnp 00:08 0800-0803 : ACPI PM1a_EVT_BLK 0804-0805 : ACPI PM1a_CNT_BLK 0808-080b : ACPI PM_TMR 0810-0815 : ACPI CPU throttle 0820-0820 : ACPI PM2_CNT_BLK 0828-082f : ACPI GPE0_BLK 0cf8-0cff : PCI conf1 d000-dfff : PCI Bus 0000:06 d000-d0ff : PCI CardBus 0000:07 d400-d4ff : PCI CardBus 0000:07 d800-d8ff : 0000:06:07.0 d800-d8ff : 8139too e400-e41f : 0000:00:1d.3 e400-e41f : uhci_hcd e480-e49f : 0000:00:1d.2 e480-e49f : uhci_hcd e800-e81f : 0000:00:1d.1 e800-e81f : uhci_hcd e880-e89f : 0000:00:1d.0 e880-e89f : uhci_hcd ec00-ec07 : 0000:00:02.0 ffa0-ffaf : 0000:00:1f.2 ffa0-ffaf : ata_piix
/proc/iomem 00000000-0000ffff : reserved 00010000-0009fbff : System RAM 0009fc00-0009ffff : reserved 000a0000-000bffff : Video RAM area 000c0000-000c7fff : Video ROM 000cf000-000cffff : Adapter ROM 000e0000-000fffff : reserved 000f0000-000fffff : System ROM 00100000-3f7bffff : System RAM 00100000-00575c47 : Kernel code 00575c48-0078d407 : Kernel data 0081a000-008a809f : Kernel bss 3f7c0000-3f7cdfff : ACPI Tables 3f7ce000-3f7fffff : ACPI Non-volatile Storage 3f800000-3fffffff : RAM buffer 40000000-43ffffff : PCI Bus 0000:06 40000000-43ffffff : PCI CardBus 0000:07 44000000-440fffff : PCI Bus 0000:03 44000000-440fffff : PCI Bus 0000:04 44000000-44003fff : 0000:04:00.0 44004000-440047ff : 0000:04:00.0 44004000-440047ff : ohci1394 48000000-4bffffff : PCI CardBus 0000:07 d0000000-dfffffff : 0000:00:02.0 e0000000-e3ffffff : PCI MMCONFIG 0 [00-3f] e0000000-e3ffffff : pnp 00:0b fec00000-fec00fff : IOAPIC 0 fec18000-fec1ffff : pnp 00:0a fec20000-fec27fff : pnp 00:0a fed00000-fed003ff : HPET 0 fed13000-fed19fff : pnp 00:01 fed1c000-fed1ffff : pnp 00:08 fed20000-fed3ffff : pnp 00:08 fed40000-fed8ffff : pnp 00:08 fee00000-fee00fff : Local APIC fee00000-fee00fff : reserved fee00000-fee00fff : pnp 00:0a ff100000-ff1fffff : PCI Bus 0000:02 ff1f0000-ff1fffff : 0000:02:00.0 ff1f0000-ff1fffff : ath5k ff300000-ff3fffff : PCI Bus 0000:06 ff300000-ff300fff : 0000:06:01.0 ff300000-ff300fff : yenta_socket ff3ffc00-ff3ffcff : 0000:06:07.0 ff3ffc00-ff3ffcff : 8139too ff580000-ff5fffff : 0000:00:02.1 ff63bc00-ff63bfff : 0000:00:1d.7 ff63bc00-ff63bfff : ehci_hcd ff63c000-ff63ffff : 0000:00:1b.0 ff63c000-ff63ffff : ICH HD audio ff640000-ff67ffff : 0000:00:02.0 ff680000-ff6fffff : 0000:00:02.0 ff780000-ffffffff : reserved ffb00000-ffbfffff : pnp 00:08 fff80000-ffffffff : pnp 00:08
7.5. PCI information attached (pci.info)
7.6. /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: ATA Model: TOSHIBA MK1237GS Rev: DL13 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi1 Channel: 00 Id: 00 Lun: 00 Vendor: PIONEER Model: DVD-RW DVR-K17A Rev: 3.52 Type: CD-ROM ANSI SCSI revision: 05 Host: scsi4 Channel: 00 Id: 00 Lun: 00 Vendor: JMicron Model: HDD Rev: Type: Direct-Access-RBC ANSI SCSI revision: 04 [unhandled content-type:application/x-info] | |