How to deal with bad valgrind output?

[Red Squirrel]

I am trying to find a memory leak in one of my apps, as far as code goes it does not look like there is one anywhere but according to valgrind there is.

Problem is it's giving me line numbers that make absolutely no sense, it's not pointing to any locations where memory is allocated. It's not even pointing to places in my program but rather to places within STL libraries which I'm sure don't have any bugs in them.

Is there a way to get better output or somehow pin point better to where the error actually is?

Some of the errors do point towards places where I use new, but there IS an appropriate delete that gets called down the line. Normally I follow the rule that whatever class allocates memory must deallocate it too, and usually this happens in the destructor.

Also what is the most accurate representation of how much memory my app is using when using ps aux? There's VSZ and RSS which I think are different types of memory, but not sure what number I should be looking at. I'm starting to wonder if valgrind is getting confused with the way I'm handling my memory or something and outputting false positives, so best bet is to probably just monitor it manually. If the number does not keep going up then technically I have no leak.

 

[Crusty]

Without seeing some code and/or messages from valgrind it's going to be difficult to help with that. I would make sure to compile with all optimizations turned off and with debugging info enabled.

Unless you're writing some really convoluted code I doubt valgrind is throwing up false positives, and even then I'm not sure it's going to matter what your code looks like if it has inherent memory management errors. How are you invoking valgrind?

As for your memory questions, as a simplified version.... VSZ is the size of virtual memory your process is using, RSS is the size residing in system RAM.

 

[Red Squirrel]

It's quite large, but here is the code:

http://redsquirrel.me/misc/envirostate.tar.gz

This is how it is being invoked:

valgrind -v --track-origins=yes --leak-check=full ./envirostate -agent -debuglevel=5

==4773== Memcheck, a memory error detector
==4773== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==4773== Using Valgrind-3.6.0 and LibVEX; rerun with -h for copyright info
==4773== Command: ./envirostate -agent -debuglevel=5
==4773== 
--4773-- Valgrind options:
--4773--    -v
--4773--    --track-origins=yes
--4773--    --leak-check=full
--4773-- Contents of /proc/version:
--4773--   Linux version 2.6.32-220.2.1.el6.centos.plus.x86_64 (mockbuild@c6b18n3.bsys.dev.centos.org) (gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC) ) #1 SMP Thu Dec 22 23:32:31 GMT 2011
--4773-- Arch and hwcaps: AMD64, amd64-sse3-cx16
--4773-- Page sizes: currently 4096, max supported 4096
--4773-- Valgrind library directory: /usr/lib64/valgrind
--4773-- Reading syms from /data/envirostate/dev/envirostate (0x400000)
--4773-- Reading syms from /usr/lib64/valgrind/memcheck-amd64-linux (0x38000000)
--4773--    object doesn't have a dynamic symbol table
--4773-- Reading syms from /lib64/ld-2.12.so (0x3a64000000)
--4773-- Reading suppressions file: /usr/lib64/valgrind/default.supp
--4773-- REDIR: 0x3a64017460 (strlen) redirected to 0x38042ae7 (vgPlain_amd64_linux_REDIR_FOR_strlen)
--4773-- Reading syms from /usr/lib64/valgrind/vgpreload_core-amd64-linux.so (0x4801000)
--4773-- Reading syms from /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so (0x4a02000)
==4773== WARNING: new redirection conflicts with existing -- ignoring it
--4773--     new: 0x3a64017460 (strlen              ) R-> 0x04a07830 strlen
--4773-- REDIR: 0x3a640172d0 (index) redirected to 0x4a07470 (index)
--4773-- REDIR: 0x3a64017350 (strcmp) redirected to 0x4a07df0 (strcmp)
--4773-- Reading syms from /usr/lib64/libstdc++.so.6.0.13 (0x3a6b000000)
--4773--    object doesn't have a symbol table
--4773-- Reading syms from /lib64/libm-2.12.so (0x3a64c00000)
--4773-- Reading syms from /lib64/libgcc_s-4.4.6-20110824.so.1 (0x3a6ac00000)
--4773--    object doesn't have a symbol table
--4773-- Reading syms from /lib64/libc-2.12.so (0x3a64400000)
--4773-- REDIR: 0x3a64483980 (strcasecmp) redirected to 0x4801560 (_vgnU_ifunc_wrapper)
--4773-- REDIR: 0x3a64485c40 (strncasecmp) redirected to 0x4801560 (_vgnU_ifunc_wrapper)
--4773-- REDIR: 0x3a644818f0 (__GI_strrchr) redirected to 0x4a072f0 (__GI_strrchr)
--4773-- REDIR: 0x3a6447fe10 (__GI_strlen) redirected to 0x4a077f0 (__GI_strlen)
--4773-- REDIR: 0x3a6447e390 (strcmp) redirected to 0x4801560 (_vgnU_ifunc_wrapper)
--4773-- REDIR: 0x3a64524ca0 (__strcmp_sse42) redirected to 0x4a07d50 (strcmp)
--4773-- REDIR: 0x3a6447fdd0 (strlen) redirected to 0x4801560 (_vgnU_ifunc_wrapper)
--4773-- REDIR: 0x3a6452ff70 (__strlen_sse42) redirected to 0x4a077d0 (strlen)
--4773-- REDIR: 0x3a6b0bd0a0 (operator new(unsigned long)) redirected to 0x4a06bc0 (operator new(unsigned long))
--4773-- REDIR: 0x3a64488320 (memcpy) redirected to 0x4a07ec0 (memcpy)
--4773-- REDIR: 0x3a64482000 (memchr) redirected to 0x4a07e90 (memchr)
--4773-- REDIR: 0x3a64479760 (malloc) redirected to 0x4a05f10 (malloc)
--4773-- REDIR: 0x3a6b0bd1d0 (operator new[](unsigned long)) redirected to 0x4a06680 (operator new[](unsigned long))
--4773-- REDIR: 0x3a6b0bb350 (operator delete[](void*)) redirected to 0x4a04d90 (operator delete[](void*))
--4773-- REDIR: 0x3a6447a590 (free) redirected to 0x4a05890 (free)
--4773-- REDIR: 0x3a6b0bb310 (operator delete(void*)) redirected to 0x4a05390 (operator delete(void*))
--4773-- REDIR: 0x3a64482080 (bcmp) redirected to 0x4801560 (_vgnU_ifunc_wrapper)
--4773-- REDIR: 0x3a6453b010 (__memcmp_sse4_1) redirected to 0x4a08a50 (bcmp)



(snip app running / debug stuff - too long to post)

(kill app by deleting pid file)

[Oct-12-2012 01:57:28pm] (5)             localhost: TcpSend: wfin
[Oct-12-2012 01:57:28pm] (5)             localhost: TcpRecv: 
6F 6B 
o  k  
[Oct-12-2012 01:57:28pm] (1) localhost: Disconnected from server
[Oct-12-2012 01:57:28pm] (1) Envirostate stopped gracefully
--4773-- Discarding syms at 0x50531f0-0x505b648 in /lib64/libnss_files-2.12.so due to munmap()
==4773== 
==4773== HEAP SUMMARY:
==4773==     in use at exit: 3,035 bytes in 66 blocks
==4773==   total heap usage: 143,758 allocs, 143,692 frees, 31,983,747 bytes allocated
==4773== 
==4773== Searching for pointers to 66 not-freed blocks
==4773== Checked 179,144 bytes
==4773== 
==4773== 27 bytes in 1 blocks are definitely lost in loss record 5 of 26
==4773==    at 0x4A06C8E: operator new(unsigned long) (vg_replace_malloc.c:261)
==4773==    by 0x3A6B09C3C8: std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) (in /usr/lib64/libstdc++.so.6.0.13)
==4773==    by 0x3A6B09CFB4: ??? (in /usr/lib64/libstdc++.so.6.0.13)
==4773==    by 0x3A6B09D118: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&, unsigned long, unsigned long) (in /usr/lib64/libstdc++.so.6.0.13)
==4773==    by 0x3A6B09D151: std::string::substr(unsigned long, unsigned long) const (in /usr/lib64/libstdc++.so.6.0.13)
==4773==    by 0x4061DD: rslib::ClArgs::DefineProgramPath() (args.cpp:247)
==4773==    by 0x40470F: rslib::ClArgs::ClArgs(int, char**, std::string) (args.cpp:28)
==4773==    by 0x42C1A8: main (envirostate.cpp:51)
==4773== 
==4773== 36 bytes in 1 blocks are definitely lost in loss record 10 of 26
==4773==    at 0x4A06C8E: operator new(unsigned long) (vg_replace_malloc.c:261)
==4773==    by 0x3A6B09C3C8: std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) (in /usr/lib64/libstdc++.so.6.0.13)
==4773==    by 0x3A6B09CFB4: ??? (in /usr/lib64/libstdc++.so.6.0.13)
==4773==    by 0x3A6B09D118: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&, unsigned long, unsigned long) (in /usr/lib64/libstdc++.so.6.0.13)
==4773==    by 0x3A6B09D151: std::string::substr(unsigned long, unsigned long) const (in /usr/lib64/libstdc++.so.6.0.13)
==4773==    by 0x4062AD: rslib::ClArgs::DefineProgramName() (args.cpp:260)
==4773==    by 0x404772: rslib::ClArgs::ClArgs(int, char**, std::string) (args.cpp:29)
==4773==    by 0x42C1A8: main (envirostate.cpp:51)
==4773== 
==4773== 36 bytes in 1 blocks are definitely lost in loss record 11 of 26
==4773==    at 0x4A06C8E: operator new(unsigned long) (vg_replace_malloc.c:261)
==4773==    by 0x3A6B09C3C8: std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) (in /usr/lib64/libstdc++.so.6.0.13)
==4773==    by 0x3A6B09CFB4: ??? (in /usr/lib64/libstdc++.so.6.0.13)
==4773==    by 0x3A6B09D118: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&, unsigned long, unsigned long) (in /usr/lib64/libstdc++.so.6.0.13)
==4773==    by 0x3A6B09D151: std::string::substr(unsigned long, unsigned long) const (in /usr/lib64/libstdc++.so.6.0.13)
==4773==    by 0x4049BE: rslib::ClArgs::ClArgs(int, char**, std::string) (args.cpp:59)
==4773==    by 0x42C1A8: main (envirostate.cpp:51)
==4773== 
==4773== 123 (32 direct, 91 indirect) bytes in 1 blocks are definitely lost in loss record 20 of 26
==4773==    at 0x4A06C8E: operator new(unsigned long) (vg_replace_malloc.c:261)
==4773==    by 0x435E80: __gnu_cxx::new_allocator<rslib::ClArgs::ArgList>::allocate(unsigned long, void const*) (new_allocator.h:89)
==4773==    by 0x433046: std::_Vector_base<rslib::ClArgs::ArgList, std::allocator<rslib::ClArgs::ArgList> >::_M_allocate(unsigned long) (stl_vector.h:140)
==4773==    by 0x4318B1: rslib::ClArgs::ArgList* std::vector<rslib::ClArgs::ArgList, std::allocator<rslib::ClArgs::ArgList> >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<rslib::ClArgs::ArgList const*, std::vector<rslib::ClArgs::ArgList, std::allocator<rslib::ClArgs::ArgList> > > >(unsigned long, __gnu_cxx::__normal_iterator<rslib::ClArgs::ArgList const*, std::vector<rslib::ClArgs::ArgList, std::allocator<rslib::ClArgs::ArgList> > >, __gnu_cxx::__normal_iterator<rslib::ClArgs::ArgList const*, std::vector<rslib::ClArgs::ArgList, std::allocator<rslib::ClArgs::ArgList> > >) (stl_vector.h:963)
==4773==    by 0x42EF09: std::vector<rslib::ClArgs::ArgList, std::allocator<rslib::ClArgs::ArgList> >::operator=(std::vector<rslib::ClArgs::ArgList, std::allocator<rslib::ClArgs::ArgList> > const&) (vector.tcc:165)
==4773==    by 0x42DA72: rslib::ClArgs::operator=(rslib::ClArgs const&) (args.h:40)
==4773==    by 0x41B5B0: AgentCore::AgentCore(rslib::ClArgs, AppLogger*) (AgentCore.cpp:15)
==4773==    by 0x42CD16: main (envirostate.cpp:105)
==4773== 
==4773== 811 (72 direct, 739 indirect) bytes in 1 blocks are definitely lost in loss record 25 of 26
==4773==    at 0x4A06C8E: operator new(unsigned long) (vg_replace_malloc.c:261)
==4773==    by 0x41C1AE: AgentCore::Load() (AgentCore.cpp:94)
==4773==    by 0x42D02F: main (envirostate.cpp:128)
==4773== 
==4773== 2,002 (128 direct, 1,874 indirect) bytes in 1 blocks are definitely lost in loss record 26 of 26
==4773==    at 0x4A06C8E: operator new(unsigned long) (vg_replace_malloc.c:261)
==4773==    by 0x43AEC6: __gnu_cxx::new_allocator<MonitorEntry*>::allocate(unsigned long, void const*) (new_allocator.h:89)
==4773==    by 0x439DCC: std::_Vector_base<MonitorEntry*, std::allocator<MonitorEntry*> >::_M_allocate(unsigned long) (stl_vector.h:140)
==4773==    by 0x43787C: std::vector<MonitorEntry*, std::allocator<MonitorEntry*> >::_M_insert_aux(__gnu_cxx::__normal_iterator<MonitorEntry**, std::vector<MonitorEntry*, std::allocator<MonitorEntry*> > >, MonitorEntry* const&) (vector.tcc:322)
==4773==    by 0x4349C4: std::vector<MonitorEntry*, std::allocator<MonitorEntry*> >::insert(__gnu_cxx::__normal_iterator<MonitorEntry**, std::vector<MonitorEntry*, std::allocator<MonitorEntry*> > >, MonitorEntry* const&) (vector.tcc:126)
==4773==    by 0x431CAD: rslib::xVector<MonitorEntry*>::Add(MonitorEntry*, int) (containers.cpp:182)
==4773==    by 0x42F269: rslib::xVector<MonitorEntry*>::Add(MonitorEntry*) (containers.cpp:189)
==4773==    by 0x41E262: AgentCore::RecvMonitors() (AgentCore.cpp:355)
==4773==    by 0x41D252: AgentCore::Poll() (AgentCore.cpp:216)
==4773==    by 0x42D1FB: main (envirostate.cpp:138)
==4773== 
==4773== LEAK SUMMARY:
==4773==    definitely lost: 331 bytes in 6 blocks
==4773==    indirectly lost: 2,704 bytes in 60 blocks
==4773==      possibly lost: 0 bytes in 0 blocks
==4773==    still reachable: 0 bytes in 0 blocks
==4773==         suppressed: 0 bytes in 0 blocks
==4773== 
==4773== ERROR SUMMARY: 6 errors from 6 contexts (suppressed: 6 from 6)
--4773-- 
--4773-- used_suppression:      6 dl-hack3-cond-1
==4773== 
==4773== ERROR SUMMARY: 6 errors from 6 contexts (suppressed: 6 from 6)

Anything that is in rslib namespace is library that I coded and use for tons of other programs and it has been tested heavily. xVector is what is used the most in this app and is basically just a wrapper library to make certain tasks with vectors easier to use. It's in containers.h/.cpp

 

[Crusty]

When you create a new object on the heap and store it's address in a pointer of it's base class's type and you do not declare the destructor to be virtual in the base class then your derived classes destructor will never be called.

So because your Core class does not have a virtual destructor the destructors for ServerCore or AgentCore will never be called when you delete the pointer.

 

[Red Squirrel]

Woah really? I did not know this. Does this go for all classes that get used as a pointer, or just classes that have inheritance like Core?

I will give that a try and report back. I've restructured a lot of stuff, I found AgentCore and ServerCore had lot of redundant code which I moved to Core so once all this is verified and tested I'll change it to virtual and try to run valgrind again and see what happens.

 

[Red Squirrel]

Sweet well that worked! I had a few errors to fix related to newer changes I made but now I'm getting zero errors. I still have to try different configurations/situations to double check everything but it's looking good.

I guess it makes sense that it would have to be virtual, since when I'm deleting the base it's not aware of the derived destructor but making it virtual would make sure it knows. Just never even thought of checking if the destructor was actually executing, that should have been my first test. Just assumed it was.

Thanks for the help, very much appreciated. One thing less that will bother me now. I had ran it all night and it never leaked anything as the issue was only with the main class, but it still bothered me to not have a clean exit.

 

[Crusty]

Sweet well that worked! I had a few errors to fix related to newer changes I made but now I'm getting zero errors. I still have to try different configurations/situations to double check everything but it's looking good.

I guess it makes sense that it would have to be virtual, since when I'm deleting the base it's not aware of the derived destructor but making it virtual would make sure it knows. Just never even thought of checking if the destructor was actually executing, that should have been my first test. Just assumed it was.

Thanks for the help, very much appreciated. One thing less that will bother me now. I had ran it all night and it never leaked anything as the issue was only with the main class, but it still bothered me to not have a clean exit.

That is precisely what unit and integration tests are for.