I have a Matillion instance with 16GB of memory, and Datadog says it is at a constant utilization rate of 95-100% (15GB+).
Q: Is that just really good memory utilization (by using everything available) or are we hitting the ceiling and I need more memory?
Hi @dodonnell
Apologies for this not being picked up sooner, I have raised this query internally and will share an update with you when I get one, in the meantime if you need anything else please do just let me know.
Kind regards, Joe
Hello @dodonnell
Are you using version 1.68 / 1.72 onwards? You can enable performance Monitor within Emerald.properties if you add: FEATURE_SWITCHES=PERFORMANCE_MONITOR
Then restart and it should show your consumption from METL.
METL would crash if it cannot commit anymore memory too, so if you are running fine and seeing no issues, it seems everything is working well.
Our regpacks have 2 GB out of 8 GB total 'committed', so the other ~70% of the memory being used is being used as a cache, usually either Linux filesystem cache or Postgres persistence cache. Which is good it means everything will run quicker when needed. Memory usage tends to approach 100% over time for this reason; it's being 'used', but it can be freed up at a moment's notice for more useful purposes for example.
I hope that helps, do let me know if you have any more questions.
Kind regards, Joe
TY for responding. I was wondering if these forums were even monitored.
We just upgraded to 1.68 last month.
What should I be looking for in the PERFORMANCE_MONITOR and how would I access the metrics?
Also, to reiterate my original question, would it be normal to be at 95% utilization? Is that expected?
