847-505-9933 | +91 20 66446300 info@datametica.com

Big Data Support Services

Beyond education, installation and execution, we offer par outstanding Big Data Support. Our technical engineers are highly qualified, with their collective knowledge covering every component of Big Data installation. You will be able to tap into their experience, and consult them on relevant issues. Depending on the level of support you need, you can choose from our various support models. With DataMetica support on your side, you are sure to get the best out of your Big Data Hadoop implementation.

Basic Ticket Support

This is a regular support model which follows the problem-solution methodology. It is a reactive support solution, and DataMetica experts get involved only when there is an issue. This support service is available on a 24 x 7 model with a built in SLA to ensure continuity of operations.

 

The basic components of this offering are:

  • Troubleshoot and manage bug fixes and patches on the Hadoop platform.
  • Troubleshoot OS/Hadoop Software returned error messages.
  • Troubleshoot Hadoop Configuration issues
  • Troubleshoot system issues
  • Performance Tuning of Clusters
  • Routine Maintenance
  • Knowledge-based articles
  • Feature requests

Hadoop Platform Support

DataMetica’s support engineers are equipped to handle any kind of issue, having collectively managed several petabytes of data on the Hadoop platform. Our platform administrators will proactively work towards supporting the Hadoop platform in your enterprise. Specifically, these are the areas our Hadoop Platform Administration services will support you in.
Cluster maintenance - Creation, removal and adding-back blacklisted nodes
  • Increasing the cluster capacity to manage the increasing load in an ongoing task.
  • For a multi-node Hadoop cluster it is not uncommon for some nodes to go down. This may be due to a software, hardware or application issue.
  • It is possible that the entire cluster performance goes down due to few faulty nodes.
  • A proactive monitoring of Hadoop cluster to avoid capacity issues, performance degrade or failures.
Plan and execute maintenance downtime
  • For Hadoop clusters, partial downtime is required for upgrades and housekeeping activities. For pre 2.0 releases, a full downtime is required in case of Job Tracker maintenance/upgrade. The support team will plan and manage this downtime.
System integration – Manage connections to source and target systems
There are the various methods of data ingress and egress in Hadoop.
  • Sqoop – for bulk load transfers
  • FTP
  • DB utilities like Oracle unload, Teradata Fast Export
  • Flume
These integrations are created and managed by the support engineering team.
Troubleshoot system integration and data integration issues
As Hadoop interacts with multiple and diverse source and target systems, several variables are at play, e.g., delay at source, unavailability of target system, issue with network, and quality of input feed, etc.  The support team needs to monitor, isolate and troubleshoot these issues working closely with other teams.
Plan, Implement and manage security, including user ID management
  • Implement and manage the security/permission model as designed by the architecture group.  This permission model for files and directories is similar to UNIX. This involves managing ongoing requests for grant/revoke privileges.
  • Critically manage and control who can do what. Though Hadoop has robust security implemented at the file system level, a stronger authentication model and a more granular authorization mechanism are required.
  • Using Kerberos which provides a fully secure cluster by leveraging the existing enterprise user-store (LDAP/active directory). We will also integrate Sentry which is used in the Hadoop ecosystem to provide a more granular, role-based authorization for users to access data at the right level of details.
Monitoring of Hadoop cluster health – data nodes and master nodes
Hadoop cluster health is continuously monitored by the support team to proactively solve a potential problem.  In addition to monitoring overall hardware, software and network, this also involves keeping a close watch on components like Name Node, Data Nodes, Job Tracker, Task Tracker etc.
HDFS maintenance including space management
  • Keeping a close track of the HDFS capacity for un-interrupted operations. During the early phase of inception when purge policies are very loosely defined, it is very common for different teams to blot the capacity even for a big cluster.
  • Hadoop-like systems are also used a lot by data analysts/data scientists for ad hoc queries. Their use-cases would mostly require a tremendous amount of HDFS capacity.

Our team will mitigate the space risk by successfully deploying various tools/practices like periodic audit and clean up, replication factor, compression etc.

Manage NN namespace backups and respond to Hadoop log files errors
Managing Name node name space backups is one of the most critical activities of the support team. These backups need to be periodically restored and validated. In addition, support is required to regularly scan for errors in Hadoop logs and take necessary action.
Automation Tools and their support
For smooth operations, we have utilities for automating bulk of work. These utilities can be generic and also very specific to a particular customer’s needs. Creation, deployment and ongoing support of these tools require in-depth knowledge of the technology, Hadoop and customer use cases.
Write and maintain knowledge base articles
  • The support team will maintain exhaustive issue logs. Ideally, these need to be reviewed during every shift handover.
  • The support team will publish knowledge based articles related to issue resolution, special case handling, new insight, automation etc.
Training – Basic Hadoop how-to for developers and administrators
The Hadoop support team will train your community of users in the following areas:

  • Getting started
  • Proper usage of the cluster resources; Do’s and Don’ts
  • Best practices – Tips and tricks
  • Modifications in the cluster environment

Hadoop Application Support

To get the best value out of your Big Data Hadoop installation, and gain the high returns that Big Data solutions promise, you need to keep your applications up-to-date and functional. DataMetica’s broad capabilities around Hadoop applications working on the most demanding workloads, will certainly add value to you.
7*24 application monitoring – Identify and alert/resolve job failures
After the successful hand-over from the delivery team, the support team will identify, troubleshoot, isolate and resolve various issues. These issues/warnings can be due to system problems, run window, data quality etc.
Plan and manage application maintenance, code migration and version updates
The support team works closely with the delivery team during code deployment and application maintenance to create and manage the environment. These are the activities the support team will concentrate on:
  • Create production directory structures
  • Create application users and provide permissions
  • Run ad hoc requests. E.g. Initial data load, configuration change etc
  • Sanity check
  • Clean up, purge, compression
Performance tuning of applications
  • The programming model used in Hadoop is MapReduce. PIG Latin is the preferred language for writing data processing code. Though it is fairly simple to code in PIG Latin, it still takes some experience and mentoring to create a performance optimized code.
  • Performance of an application can go down over a period of time due to various reasons like capacity/bandwidth, growth in data, network etc. The support team will be involved hands-on in troubleshooting, isolation and resolution of these issues.
Publish production application monitoring report
This is a single-window view of all the jobs running on your systems. This report gives a snapshot of the production run detailing the jobs that are successfully completed, those which gave some warnings and those that failed. These details are used for resolving current issues, early identification of any major breakdown and cluster planning decisions.
Application Code bug fix and minor enhancements (less than 1 week)
The support team will handle small bug fixes or minor enhancements which do not merit creating a development team. For example, adding a flag in a particular table.  Support team can help in implementing similar small tasks.
Facilitate Production support and change approval meetings
The production support team is responsible for uninterrupted Hadoop cluster operations. This team will give a Go-No Go decision to all production deployments. The support team needs to either run these change approval meetings or be represented at these meetings.
Documentation
During the transition of applications to the support team, one of the key deliverables is an exhaustive run-book. The support team will own, create, validate and publish this run book for all the applications handled by them. These need a sign-off by the delivery team manager. This is also a living document which needs to be maintained by the support team.

Still something you want to know?

Have questions? Click the button on the right to drop us a line.
Contact Us

We're not around right now. But you can send us an email and we'll get back to you, asap.