Cloudera CDP How to solve Ambari Metrics corrupted data?

Sometimes Ambari Metrics stops displaying data in the dashboards.

Or You may be getting time out issues that were not solved (see

https://community.hortonworks.com/questions/4726/ambari-metric-collector-error-sending-metric-to-se....)

At other times, especially on a new install you may receive this error in the Collector Log

2016-01-25 15:33:31,195 WARN 
org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.query.DefaultPhoenixDataSource:
 Unable to connect to HBase store using Phoenix.
org.apache.phoenix.schema.TableNotFoundException: ERROR 1012 (42M03): Table undefined. tableName=SYSTEM.CATALOG.

This is usually due to AMS Data being corrupt.

1. Shut down Ambari Monitors, and Collector via Ambari
2. Cleared out the /var/lib/ambari-metrics-collector dir for fresh restart
3. From Ambari -> Ambari Metrics -> Config -> Advanced ams-hbase-site get the hbase.rootdir and hbase-tmp directory

   

   

4. Delete or Move the hbase-tmp and hbase.rootdir directories to an archive folder
5. Started AMS.
6. All services will came online and graphs started to display, after a few minutes

I hereby declare my undying love for you for fixing the error I've found this morning:

Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

If I could afford it, I would by you a lifelong supply of the beverage of your choice - but I can't, so thanks shall have to suffice 

Thanks. This helped me purge metrics too as Ambari metrics services were stuck because of a disk full error. I also updated metrics retention parameters.

Ambari metrics collector connection refused : metrics data not available on Ambari Dashboard

Alert : Metrics Collector Process. Connection failed: [Errno 111] Connection refused to metrics02-node:6188

As you mentioned, i removed data from hbase.tmp.dir and hbase.rootdir.

rm -rf /var/lib/ambari-metrics-collector/hbase/*

rm -rf /var/lib/ambari-metrics-collector/hbase-tmp/*

Later restared Ambari Metrics from Ambari. It worked fine. Thanks a lot.