Introduction to Configuration Management

 Configuration Management  Comments Off on Introduction to Configuration Management
Apr 282010

Configuration management is the process that manages the resource configurations. The term “provisioning” is sometimes used interchangeably with CM.

First feature of the configuration managers is to provide abstraction, over the lower layer devices, which leads us to business process consolidation, reduced silos and increased enterprise effectiveness.

Let me give an example on this. Suppose, I have a customer order manager which manages the e2e order-to-cash business process. In a typical scenario, OM will trigger a service activator to activate the service. The SA will then go to the element managers/NEs to configure them, however, for each specific device; there may be different provisioning steps. For example, if the device is Cisco IOS based, you would probably ssh first, enter the password, then enter privileged exec mode, enter the commands. (I remember the times I was using “expect” scripts to achieve this). For another type of device, it could be done by SNMP SET requests. The examples differ. Here, via CMs GUI or NBI, I can issue the same “enable VLAN 1” command and it will be applied to multiple types of devices by the CM.

The second feature of the CMs is configuration tracking. CMs connect to the devices on scheduled intervals and pull the configuration data. They put this data in their repository. This behavior enables us to rollback to a previous configuration in the case of a failure.

CMs also provide user-friendly Diff in order to highlight you which parts of the configuration have been changed.

The third feature is to enforce the policies. Organizations are obligated to conform to specific policies and those should be reflected to the configurations. For example, for a router, disabled telnet access (they want to use ssh) requirement can be implemented on the CM. This case, CM will not allow this request to be passed to the devices, providing a control mechanism.

Individuals may bypass the CM and go and configure the systems directly. This should be considered as an extra-ordinary situation. However, the CMs will also help you to detect discrepancies in the configuration data in their next configuration polling and generate notifications (via trap, email etc.)

We see 3 types of configuration managers in today’s telecommunications environment:

– Server Configuration Managers – manage the servers/PCs. They install OS, install applications, and apply patches.

– Network Configuration Managers – manage firewalls, IPS/IDS, Routers, switches. Their configurations, IOS updates, bulk configurations etc.

– Storage Configuration Managers – manage the storage systems (SAN, NAS etc.)

Most of the devices have element management systems that do the configuration management function, and they are very successful in this. However, these element management systems can only manage specific types of elements. So, if your infrastructure is a multi-vendor environment involving multiple types of devices, investing on a configuration management will definitely bring efficiency to your organization.

Inventory Management – Network Discovery

 Inventory Management  Comments Off on Inventory Management – Network Discovery
Apr 182010

Network discovery (or auto-discovery) is the process that automatically populates the physical and logical inventory information of a given network. Without auto-discovery, all the inventory items would be entered to the NIMS (Network Inventory Management System) manually, or at most, be imported from a previously generated flat file. Manual entries bring so many burdens especially if your network is dynamic and each day you add/remove new devices, new cards and provision new circuits.

A note in here: Auto-discovery processes may not only be implemented by the Inventory Management Systems, but also other OSS systems such as Fault Managers, Performance Managers or SQMs which need to employ the network topology information. In an ideal world, all the inventory information should be kept in one place, in the NIMS. However, to be able to sell the individual types of OSS products separately, vendors needed to include an inventory module within their applications. This introduces some scenarios in which multiple OSS products try to discover the network at the same time which may lead to inefficiencies in the network performance.

Auto-discovery of a network includes 3 phases:

• Detection
• Device Discovery
• Topology Creation

Detection phase identifies all the “living” devices on the network. This is achieved via several mechanisms.

The most popular one is the ICMP Sweeping (or pinging). In this method, you provide an IP pool (typically a subnet) to the tool that will do the discovery. You may also have some options to exclude some IP addresses within this pool. This is because, for some types of devices/interfaces (such as ISDN dial-backup interfaces) unnecessary traffic (even ping) may result in connection charges. Another reason may be limiting the management traffic on a highly utilized interface.

For big networks the discovery process can take days. So, you should split the network into multiple small ones to increase the discovery efficiency and reduce discovery time.

After the IP address set is constructed, the tool starts to ping those IP addresses. If it receives a response, it notes this down.

You may also have the option to import the IP information from a DNS server. If you are using DNS server effectively, this method will provide you all the “authorized” IPs in your network. One other way could be reading the ARP cache of a device, but for me ARP is not very reliable. (The cache is cleared when the device restarts and some items in the cache are subject to time-out after a certain period of time)

The second step in auto-discovery is the Device Discovery. In this step, the discovery engine will try to connect to a found device via specified management protocols (SNMP, TL1 etc.) to receive additional inventory information. Ideally, this information will include all the inventory information for that NE (Network Element). How this is done differs from protocol to protocol. I will try to explain the SNMP device discovery which seems to be the most adapted one.

All devices that implement SNMP management protocol, should implement the MIB-2 RFC. This MIB (Management Information Base) was designed to include generic information that fits to all vendor equipments such as Name, Description, and Interfaces etc. However, most vendors prefer to use their own propriety MIB trees. (Which are located under the enterprise branch of the OID hierarchy). The motivation differs. Some vendors want to “hide” their devices to force the external OSS systems, to interface with the vendor EMS’s NBIs. Some other vendors may just want to distribute the MIBs to registered users for marketing purposes.

Vendors (whether they provide their own MIBs or not) must fill in some MIB-2 values for their NEs. The most important ones for the discovery purposes are the sysObjID, sysName and sysLocation. SysName and sysLocation will be very beneficial values to have, if they are correctly implemented.

SysObjID is the one which helps the discovery engine to understand the device type (host, switch etc.), vendor name, and model name. All the SNMP enabled vendors should register their devices to IANA organization, in order to get a SysObjID. SysObjID is unique in the world, just like a MAC address. This uniqueness enables the management system to map a SysObjID to a specific device type, vendor name and model.

After the device type is received from the SysObjID, further actions can be run. For example, if it is a router, discovery engine could check its MIB repository to find if there is a specific MIB for this vendor-model. (If not, it will use generic MIB-2). After the MIB is located, it will send series of SNMP-get requests to have the routing table, neighbors table, interfaces, sub-interfaces etc. If it is a switch, the discovery process will again send SNMP-get requests (to BRIDGE-MIB for example) to receive the forwarding table information.

The router example could be considered as L3 discovery, whereas the switch example is an example of a L2 Ethernet discovery. Other types of L2 discoveries can be for ATM, Frame Relay and MPLS type of networks.

At the final phase, topology creation, the discovered NE information is correlated by the inventory process to find and visualize the physical and logical connections between the NEs. Topology information can also be fetched directly from devices. For example OSPF protocol mandates the devices to maintain the topology information. Propriety protocols such as Cisco CDP can also be utilized by the discovery processes to draw the network topology.

Apr 122010

SID (Shared Information and Data model) is one of the frameworks of the TMForum NGOSS Frameworx suite. It is also called the Information Framework. SID can be considered as the language of the NGOSS. It helps us to identify the business entities that play role in the business processes of a telecommunications service provider.

As I mentioned in my previous eTOM article, eTOM is the framework that identifies the business processes of a service provider. When modeling the business processes, we have to involve some business entities. For example, suppose, in an end-to-end fulfillment process, a customer selects a product offering from the product catalog to buy at a specified price. During his/her interaction with the order fulfillment system, he/she also specifies some attributes of the product he/she is buying (such as the color of the physical resource that will be delivered). Along with this resource, and as part of the product, he/she also buys a service (a communication service such as DSL). The process goes on until the customer order is successfully delivered to the customer.

Some of the entities that play role in this specific scenario can be listed as customer, product catalog, product, service, resource, price, offering, order and interaction. All of these entities are the subjects of the SID framework.

SID helps you to find out the domain objects and their relationships between them when you are designing an NGOSS solution. Business domain analysis is a very time consuming and complex step in system analysis & design phase. It is complex because you need to know the domain very well. (That’s why we require analysts in the projects).During the analysis, you should identify the entities that should appear in the domain. The analysis should also include the relationships and dependencies between those entities. For a telecommunications provider, luckily, this analysis is ready for us to use. SID provides a ready-to-use domain model, based on the best practices, which can be directly applied to the new or re-engineering projects. Using SID in the analysis & design phases saves us a lot of time.

SID is the Information “and” the Data Model. Let me explain what this means. Information models are more abstract models that aim to tell us the inner workings or let’s say the nature of the modeled system. They look from the “business view” perspective. The data model, on the other hand, appears when you make this information more concrete. (When you apply it as a database system or a class diagram) In the information model, I may have abstract domain objects, but in the data model, I may not prefer to implement them at all. (I may want to implement those multiple objects as one object)

I really cannot see the “data model” part in SID. The data model is the model an enterprise architect should draw and it is enterprise specific. It could be seen as the technology and application independent, “system view” perspective of the logical information model.

SID and eTOM goes hand-in-hand and TMForum provides mappings between them. SID Domains are consistent with the eTOM domains. The Domains include the Level 1 ABEs which are also mapped to eTOM Level 2 business processes. What an ABE is? Well, it is the abbreviation for Aggregate Business Entity, which means an entity that can further be decomposed to other sub-entities. (Just like the decomposition we see in the eTOM hierarchy).
SID Business Entities are modeled in UML diagrams and can be imported to UML tools. (TMForum publishes them in different importable formats). The framework also comes with an extensive documentation that explains the structure and the behaviors of the business entities. Typically, you will find an abbreviation document for each ABE.

Today, OSS application vendors are more NGOSS aware and they try to design/re-design their products based on NGOSS. In order for two different OSS systems to communicate effectively, their data structures should be consistent with each other. This way, it will require minimum conversion during the conversation. OSS/J enables this on the API level but the data structure of the application is also important. Using SID in designing applications eases the integrations between the different applications. Explaining your OSS product in SID terms will also be appreciated by your customers and supplier/partners.

Introduction to Mediation

 Mediation  Comments Off on Introduction to Mediation
Apr 092010

Mediation devices collect usage data in the form of xDRs (CDRs, IPDRs and EDRs etc.) from several data sources and deliver this data to specified locations in a specified format.

The devices (switches, routers, IP PDUs) and applications (IN etc.) in a telecommunication infrastructure report their usage statistics for different purposes. The main purpose is billing, so, every mediation device generally interfaces with one or more billing (or charging) systems. We may also see other integrations with Fraud Management Systems (FMS), Revenue Assurance Systems (RA), IN systems, Interconnect systems etc.

Different types of devices on the network output different types of CDRs in different formats. They also deliver those via different protocols (ftp, sftp, snmp, xfer, ftamip, s12ftp, ftmxot, netflow, odbc to name a few). Most of the CDRs are in ASCII format and some of them are in binary format (like ASN.1 and Radius)

Usage dependent applications on the other hand, want to have the usage data in a specific format. For example a billing system’s specs may say that the input data must be in ASCII format. It may also dictate the orientation of the expected file (i.e. columns within it)

Mediation devices does this protocol and file conversion. They deal with the polling logic, (network errors, retries, bad data etc.) and prepare an error-free world for the usage based applications. They detect the file errors and save them in a place where they are further analyzed and corrected. They do CDR normalization (adding prefixes, suffixes etc). They aggregate partial CDRs. They do duplication checks. You can run some rules on them, such as “discard the CDR if the usage duration is less than 5 seconds.”

Mediation devices act as a “mediator” between the systems that deal with the usage data. They act as a single point of contact. Mediation process reduces the load of other applications by taking over the collection, aggregation, normalization and conversion tasks. Considering the number of data sources and data pollers, it may require multiple load-balanced servers and dedicated organizational departments.

Performance Management – Thresholds

 Performance Management  Comments Off on Performance Management – Thresholds
Apr 062010

Thresholds are the way to detect anomalies in the performance data. They act as proactive tools for the operators to take early actions on the degradations before they cause a fault and possible service loss.

There are 2 types of thresholds.

1- ) Static or Burst thresholds
2- ) Dynamic or Baseline thresholds

The static type is the one that triggers an action whenever the specified value is crossed. Once violated, those thresholds will not trigger the same action in each new data that is above the threshold. (Most products count the occurrence times, until the value returns back to normal limits)

The second type of threshold is the one that takes into account the baseline (historical) information. These thresholds look at the baseline data to see the variations from the baseline. For example, suppose a router’s interface utilization is generally around 1% on Sunday mornings. When it becomes 10% in any Sunday morning, this should be considered a variation from the baseline.

We attach actions to the thresholds and normally these are SNMP traps. This trap has to have some attributes such as severity, additional text, managed object, probable cause etc.
Thresholds should also define a clear value which also initiates a clear trap. The clear trap will cause the alarm to be cleared on the FM side. (In order for a system to send an SNMP trap, it has to comply with a MIB. Most of the times, this MIB is enterprise specific. If you want to integrate your FM with PM over the SNMP traps, you will need the FM vendor’s MIB.)

Thresholds can be applied on raw data or aggregated data. Some products do not support the aggregated one so you have to check this important detail with your vendor.

Another important detail is the number of thresholds you deploy on your PM. Putting thousands of thresholds will eventually impact the performance of your PM. You should check with your PM vendor to understand the impacts of thresholds on the performance of the product. If it has limits (it should have limits), you may better group and classify your resources to reduce the threshold needs. In some cases you may even want to rely on RMON thresholds (the thresholds applied to and managed by the resources themselves).

Setting the right thresholds is another practice. It is organization specific and requires domain knowledge. If we don’t have a PM baseline data (we may be delivering the FM and PM in the same project), we should apply the best practice thresholds and “fine-tune” them to the organization by trial and errors.