Adaptable bandwidth planning for enhanced Qo. S support in user-centric broadband architectures Dr. Ilka Milouchewa (FHG) Dirk Hetzer (T-Systems, M&B) Dirk Hetzer, Uni Potsdam / T-Systems Seite
Bandwidth on Demand Planning. Topics n n n Background Bandwidth planning based on reinforcement learning Operations research for bandwidth scheduling Combining reinforcement learning with scheduling n Q-learning n Informed learning n Relational learning QORE system for adaptable bandwidth planning Integration of bandwidth planning in research projects and Telecom networks 1
Bandwidth on Demand Planning. Background (1) n need for planning of resources in new on-demand services (rich-media, IPTV, Vo. D, gaming) on all IP core, replacing ISDN, ATM etc. n high bandwidth demands, efficiency achievable by bandwidth planning and resource reservation Scenario: Planning based resource reservation in advance at aggregation points (DSLAM, BB-Po. P) 2
Bandwidth on Demand Planning. Background (2) Scenario: Planning is based on resource reservation “in advance” at access routers considering learning of performance data n n Need for capacity planning based on ADVANCE resource reservations for different kinds of Qo. S based applications and best effort traffic Total resources (bandwidth) are always restricted, therefore advance reservation could be used for planning and enhanced utilization n Learning of performance to predict resource requirements Optimized resource allocations adapting advance resource requests to predicted requests Find compromise between different resource requests and enhance Qo. S for all kinds of applications (traffic) 3
Convergence & Triple Play. Challanging the way of Service Creation. Content delivery on demand Access evolution path: Telephony (F/M) Others Radio Access • • • Triple Play evolution path: • • GSM GPRS EDGE 3 G W-CDMA HSDPA Broadband wireless access + DSL-/ 3 G-Internet + Vo. IP + TV + Vo. D Microwave Satellite Laser/Optical Wi. Max UNIFIED USER EXPERIENCE Fixed Line Access • • PSTN/ISDN FTTP Cable Broadband wireline access + Cable Internet + Communication Television Communications Services evolution path: SMS MMS Voice + Text + Picture + Sound Triple Play • h + DSL-Internet + Vo. IP + IPTV + Smart Home Broadband Internet Rich Media Voice + Text + Picture + Sound + Video 4
User-centric approach for bandwidth planning. Scenarios and applications Planning for Triple Play. Example 1: TV-based Telecom Services. § Integrates telephony service with a user’s television § Supports the delivery of telephony services in conjunction with cable, DSL, and IP-based video services § PSTN, mobile, or Vo. IP phones SIP Servlet Video Distribution Network PSTN /Vo. IP Bill Smith 732 -699 -3232 • TV Calling Name • User directed routing • Click-to-dial • Participation TV - Voting - Gaming - Shopping • Messaging - Picture Sharing - Multimedia Message Display - Content Services - Voice Mail Screening 5
User-centric approach for bandwidth planning. Scenarios and applications Planning for Triple Play. Example 2: TV-Mobile Convergent Participation Service (Blogging). § Participation TV as a Mobile Communication/ Broadcast Communication convergence case. § Participation TV denotes the integration of user feedback/interaction into TV-formats (such as game shows). § Traditionally very limited e. g. by calling into the game, sending SMS, … n MMS diaries (blogging) – uploads from personal pictures, texts via MMS n Virtual pubs/discos tour (spying on events) n Virtual classroom n Contests (e. g. Best amateur news report of the day/month) n MMS/i. TV chat n Other services (personalized weather reports, group contests, alerts, etc. ) 6
Bandwidth on Demand Planning. Bandwidth planning based on reinforcement learning (1) Different “learning” approaches to optimize and plan bandwidth Stochastic automata - a stochastic policy by associating a probability with each action, so that actions are chosen at random according to their probabilities - the policy does not take into consideration the current state of the system, when choosing an action. Reinforcement learning benefits -> Rewards from interactions (action) with environment at each state (dynamical learning) -> Adaptive control considering states and actions (change of bandwidth scheduling dependent on the performance “feedback”) 7
Bandwidth on Demand Planning. Bandwidth planning based on reinforcement learning (2) Scenarios for usage of reinforcement learning for bandwidth planning Learning performance (delay, throughput) and predict resource needs -> In case of HDTV Allocate resource in advance for different traffic classes -> Multimedia Conference -> GRID Transfers Adaptation of advance reservations to resource needs of traffic classes -> On-demand service -Resource usage is derived in interaction with environment - Using reinforcements, prediction is done for the period T - Optimal sharing of resources in advance for traffic classes considering reinforcements for each traffic -Different strategies for advance allocation of resources for on-demand traffic considering predicted and reinforcements of actual resource usage of traffic classes 8
Bandwidth on Demand Planning. Usage of QORE for bandwidth planning (1) Reservation in advance parameters depend on applications and users Reservation in advance Parameter Options Example Application Alternative Bandwidth Dependent on application and possibilities for varying Qo. S levels Multimedia streaming, content delivery, file download Increment Additional resource demand for traffic aggregation Vo. IP aggregate, content delivery Duration Dependent on application usage GRID, multimedia, mission critical Cost Dependent on application users (cost wanted to pay) Tele-Radiology, Vo. IP, video Time constraints Flexibility of usage: - Fixed start, - Interval based (flexible start, flexible end) - Deadline Distributed multimedia, Grid, multimedia conferencing Dependency Order of allocations to applications Grid 9
Bandwidth on Demand Planning. Usage of QORE for bandwidth planning (2) Problem: Reinforcement learning based on rewards from environment (delay of best effort traffic) -->finding optimal bandwidth allocation (optimal schedule) which satisfies resource requirements in advance of Qo. S based applications and enhancing Qo. S (delay) of best effort traffic Reinforcement learning problems for bandwidth planning -> User interface for bandwidth allocation in advance considering traffic classes -> Reinforcements: periodical performance measurements for best effort traffic -> Value function cumulative rewards evaluating the schedule for the planning period -> Interaction with Bandwidth Broker 10
Combining reinforcement learning methods and scheduling (1) Obtaining optimal bandwidth schedules for proactive and reactive planning using combination of operation research & reinforcement learning methods Pattern based scheduling Conflict-free schedule with minimum duration Partial displacement scheduling 11
Combining reinforcement learning methods and scheduling (2). Simple Q-Learning - Approach n A model-free RL approach combining conflict-free scheduling with minimum duration − n Selection of bandwidth schedule for evaluation − n Random selection, e-greedy (the best Q-value) Q-value of bandwidth schedule updated every time at the end of planning period, for instance a day − n Bandwidth schedules are characterized with Q-values based on rewards and value function (cumulative sum of rewards evaluating exceeded end-to-end delay threshold for planning periods) Update based on Learning rate (Recency Weighted Average. . ) Pure “trial” and “error” − for practical usage not efficient, because predictions of resource usage of applications are not considered 12
Combining reinforcement learning methods and scheduling (3). Simple Q-Learning - Example 13
Combining reinforcement learning methods and scheduling (9). Relational Q-Learning - Modified schedule based on patterns When pattern is detected, the planned allocation is rescheduled Using rewards from endto-end delay measurements for daily planning period 14
Bandwidth on Demand Planning. Usage of QORE for bandwidth planning (1) Integration of patterns in scheduling algorithms based on reinforcement learning Patterns - abstractions for structures describing behaviour of performance parameters for network connections - extracted from monitoring data bases Different kinds of patterns considered for bandwidth planning: - Outliers - Threshold overload patterns Bandwidth planning focus End systems Routers Network connections Immediate Traffic Routing Qo. S parameter volume event resource s request Patterns describing “normal”and “abnormal” behavior of multivariate time series data Failure events - Patterns describing traffic, Qo. S parameter behavior, routing events and anomalies of network connections 15
Bandwidth on Demand Planning. Usage of QORE for bandwidth planning (2) QORE components and interaction with monitoring and planning database QORE: automated tool for adaptable Qo. S-oriented proactive and reactive bandwidth planning Components using common knowledge database for monitoring & planning - User interface for „advance“ reservation - Qo. S parameter monitoring - Scheduling algorithms - Resource constraints simulator - Effective bandwidth estimation - Pattern analyser - Visual data mining for bandwidth planning 16
Scenario for bandwidth planning in converged fixed / mobile environment Selection of optimal access network for content delivery 17
Summary. Automated adaptable bandwidth planning in Internet based on reinforcement learning, Qo. S parameter patterns and scheduling heuristics n Novel management strategies for an automated proactive and reactive bandwidth planning in Internet using Qo. S monitoring data (QORE system) n Integration of data mining technologies using patterns for bandwidth planning n Bridging a gap in operations research techniques for bandwidth planning integrating dynamic learning of Qo. S parameters (pattern detection) n Integrated architecture based on reinforcement learning, operations research and data mining for bandwidth planning 18
Bandwidth on Demand Planning. Outlook & ongoing work in EU projects n NETQOS n n n Application specific bandwidth planning concepts Policy based bandwidth management and planning DAIDALOS n n n Enhanced Qo. S brokerage architectures based on resource reservation in advance Advance resource reservation for mobile Qo. S based services Practical integration of the adaptable bandwidth planning of QORE in network management systems 19
Скачать презентацию Adaptable bandwidth planning for enhanced Qo S support
2d8bb2643462f3d02a52b76686fc4a8e.ppt