joubin's Kardz


Microsoft Graph Security API

To build intelligent security solutions correlate alerts, get additional context to speed up investigation, automate security operations, use Microsoft Graph Security API.

  • A federated system
  • Microsoft Graph Security API
  • Security API notification subscription
  • Secure Score, system wide
  • Identity security score, AD

https://www.microsoft.com/en-us/security/intelligence-security-api





Shared by joubin on Thu Sep 27 2018

MS Ignite, Migrate to Cosmos DB

  • GA of Cassandra API
    • Table API
    • MongoDB
  • Keyspace ?
  • CQL Query Builder
  • CQL Shell (CQLSH)
  • Cassandra Stress, to verify performance
  • Scale throughput
    • Change the throughput for any table
  • SDK access
    • org.apache.spark...

Ignite 2018 session: https://myignite.techcommunity.microsoft.com/sessions/65369?source=schedule#ignite-html-anchor

Cheatsheets: https://docs.microsoft.com/en-us/azure/cosmos-db/query-cheat-sheet







Shared by joubin on Wed Sep 26 2018


NSAT

Net customer/user Satisfaction (NSAT)

NSAT is an index of net satisfaction scores across a number of segments and audiences as measured by the Global Relationship Study.

Shared by joubin on Tue Oct 02 2018

MS Ignite, Kubernetes on Azure

  • Gabe Monroy
  • Kubernetes, container orchestration
  • Portability
  • Extensible
  • Self healing
  • Kubectl
  • Etcd
  • Kubelet, heart beat back to the API server
  • Ingress
  • Kube-proxy
  • Lift and Shift application moving strategy, without refactoring your app
  • Azure managed control plane
    • Devops project, Azure resource
    • Azure Dev Spaces ?
    • Helm chart ?
    • CI/CD pipelines
    • Key Vault is used to store secrets for Kubernetes
    • Vnet ?
    • Traffic Manager, in Azure
    • ACI ?, Azure Container Instances
  • Azure Monitor for containers
  • RBAC control for fine grained Kubernetes resources access control
  • Scenarios
    • Lift and Shift to containers
    • Micro services
    • Machine Learning scenarios
    • IoT
      • Azure IoT Edge
  • Maersk
    • NSAT, customer satisfaction
  • Terraform
  • CaaS
  • OpenAI, a not for profit organization
  • Azure IoT Edge






















Shared by joubin on Tue Sep 25 2018

Onnx, Deep Learning

Open Neural Network Exchange is a format for deep learning models that allows interoperability between different open source AI frameworks.

ONNX is being co-developed as an open-source project by:

  • Microsoft
  • Amazon
  • Facebook

https://en.wikipedia.org/wiki/Onnx

Shared by joubin on Sun Sep 30 2018

MS Ignite, Event driven apps with CosmosDB

  • Event Notification pattern
  • Tight coupling prevention in the design
  • Eg adding a queue between the order module and the shipping module
  • System becomes more resilient
  • Events should have as much context around then as they can
  • Event sourcing pattern, writing the events to a log
    • Audit History
    • Decoupling
    • Materialized views
    • Integration with other systems
    • Teleporting ?
  • Microservices
  • Fault tolerance
  • Functions can be triggered based on an event
  • Azure CosmosDB
  • RU: Request Unit
  • Five consistency models ?
  • A multi model database, key value, column family, documents, graph
  • ARS, Atomic Record Sequence
  • Schema agnostic
  • Change Feed: log updates or others
    • Triggers for Azure Functions
    • Stream Processing
    • Data Movement, moving the data downstream
  • Multiple consumers of the change feed
  • Azure Functions
    • Bindings and Triggers
    • Proxies
    • Monitoring
    • Local Debugging
    • CI/CD
    • Doable Functions, will save their own states
  • Azure Event Grid
    • Event handlers: Functions, etc
  • Use the Change Feed to light up your micro services architecture
  • Jet.com uses micro services on CosmosDB and Azure Event Grid
  • Skype as well
  • CQRS
























Shared by joubin on Tue Sep 25 2018

Real-time Messaging with Azure SignalR Service

  • Roadmap
    • Q2, had public preview
    • Q3, GA
    • Q4
  • Scale with backplane with Redis etc
  • Azure Function binding, in preview
  • For GA, Free Tier and standard
  • Auto Scaling... ??
  • Scenarios
    • In app notifications
    • Live dashboard
    • Co authoring
  • It's realtime messaging
  • An Azure resource
  • dotnet build; dotnet run;
  • Azure SignalR Service on asp.net Core SignalR
  • Hub base class
  • First negotiates. If configured to use Azure SignalR service, it returns negotiation points on Azure.
  • Rest API
    • Gyro tracker demo
  • Azure Function binding for SignalR Service


































Shared by joubin on Thu Sep 27 2018

Node.js apps on Azure with VS Code

  • 12 Factor
  • Mongoose SDK Node.js sdk
  • Develop locally with MongoDB
  • Switch to CosmosDB in the cloud
    • Choose MongoDB api for CosmosDB
  • Make app cloud ready
  • dotenv package ??
  • VS Code, Ctrl+shift B, run commands










Shared by joubin on Thu Sep 27 2018

LiveTiles, QnA Maker

  • LiveTiles Bots
  • Scenarios
    • Personal Assistant
    • Team Assistant
    • Enterprise Assistant
    • External Assistant
  • What's a typical development cycle?
  • Site application as a service as a bot!!
  • QnA Maker
  • LiveTiles Bot designer
  • AIconics ?
  • EthicalOS.org
























Shared by joubin on Thu Sep 27 2018

Top Tips for Git Powered Teams

  • git push origin master:demo
    • If you forget to create a branch before pushing code remotely
  • Branch policies
    • Require PR
    • Item linking
  • Build Validation policy
    • Build must pass before merge into branch/master
  • Checkout pull requests locally ??
  • Continuous Code Quality with Automation
    • Static code analysis
      • sonarqube/sonarcloud.io
      • whitesource, security scanning
    • Drop these into pipeline
  • Git Large File Support
    • Git LFS
    • Instead of shallow cloning
    • BFG Repo Cleaner ??
  • Always use the latest version of Git
    • git update
  • Trunk-based development
    • GitHub flow is different that Release flow branching structure
    • master is never in production?? Why?
    • Cherry pick a Pull request into a new branch






















Shared by joubin on Thu Sep 27 2018

MS Ignite, Azure Pipelines and GitHub

  • Extensions
    • Slack, SonarCloud
    • Over 500 extensions
  • Integration with GitHub
  • Start with GitHub market place
    • Go to continuous integration
    • Setup Azure pipelines
    • Will land in Azure DevOps
    • Example: GitHub/microsoft/vscode
      • Has badges for Azure Pipelines
  • CredScan ?, A step that check for credentials as part of a commit


















Shared by joubin on Wed Sep 26 2018

MS Ignite, Azure Relay, Hubs, Grid, Service Bus

  • Azure Relay
    • Connect Systems
  • Event Hubs
  • Event Grid
  • Messages
    • Fact
      • Time Series, logs, streams
        • Event Hubs for streaming data
      • Discrete
        • Not time series i.e. no order to the messages
        • Event Grid, event based programming
    • Command
      • Service Bus, Enterprise messaging
  • Service Bus Explorer
  • System to system messaging
    • Contrast this with
      • Notification hub
      • SignalR hub








Shared by joubin on Wed Sep 26 2018

MS Ignite, Automated ML & Hyperparameter Tuning

  • Labeling process
  • Hyperparameters??
  • scikit learn algorithm cheat sheet
  • Automated machine learning
    • To give you a model faster
    • Submitting experiments to Automated ML
  • Jupyter notebook
  • Cross validation ??
  • KNeighbors ?
  • Set metrics, e.g. accuracy, AUC etc
  • Confusion matrix, TP TN etc
  • Configure Automated ML by setting metrics

Benefits of Automated ML

  • Model creation time, from days to hours
  • Robust benchmarking process for ML projects
  • Enable domain experts to leverage ML
  • Is an excellent starting point for selection of algorithm
  • Currently: Classification/binary, and regression

Power BI

  • Automated ML is baked into Power BI

Hyperparameter tuning

  • Adjustable parameters to given model training
  • Hyperparameter configuration
  • Find the configuration that result in best performance
  • Best accuracy
  • Number of layers in deep learning
    • This is in addition to the training datasets
  • Automated hyperparameter tuning in Azure ML
  • Early terminations of poorly running models
  • Bandit policy ??
  • This capability is available for free, for now!






























Shared by joubin on Wed Sep 26 2018

MS Ignite, Common Data Services for Apps

  • CDS, Common Data Services
  • An Azure based service
  • PowerApps portal
    • Define custom entities
    • Relationships, Lookups
    • Optionsets eg enums
    • Calculated Fields, computed columns
    • Roll-up Fields, aggregates
  • Microsoft Flows?

https://docs.microsoft.com/en-us/powerapps/maker/common-data-service/data-platform-intro

Shared by joubin on Wed Sep 26 2018

MS Ignite, Super Charged Debugging in VS 2017

Step into specific, nested methods

  • Tracepoints, Actions, pseudo debugger variables $Function
  • Inspection, nse (no side effects), i++ in the watch window won't affect the variable in the program flow itself.
    • ac, always call
  • $ReturnValue1 2 3
  • DebuggerDisplay attribute - will be used as an attribute on the class itself
    • nq, no quotes
    • h, hexidecimal
  • Make object Id, for vars that are added to a collection
  • Performance Tip
    • Intellitrace
      • Historical Debugging
    • Snapshot Debugger
      • Debugging in production???
      • Remote debugging
    • Managed heap
      • Diagnostics window, memory usage tab, take snapshot
    • CPU usage, diagnostics window
  • Format Specifiers






Shared by joubin on Wed Sep 26 2018

MS Ignite, ML.NET

  • GitHub Labeler
  • GitHub Classification issuer
  • Azure Stream Analytics, Anomaly detection
  • 0.5 is available
  • Features, Labels. Supervised machine learning
  • Model consumption
  • Transformer, takes the features and numerizing them
  • Microsoft.ML nuget package
  • Estimator is the thing to use to build your model
  • Steps
    • Create environment
    • Read data
    • Create estimator and create model
    • Build and train
    • Evaluate the model
    • Make prediction function ?
  • That's a classifier. Labeler example. Classification, however not binary.
    • Multi Classification
    • SDCA ?, A trainer.
  • Featurized text
  • Demo: movie recommendation
    • Population average, based on score
    • Content based filtering, taking into account user preference
    • Collaborative filtering
      • Factorization Machines ?
      • Movie Lens Dataset
  • Deep Learning
    • Tensorflow
    • Image classification
    • Download model
    • Microsoft.MLTensorflow nuget package
    • Microsoft.ML.ImageAnalytics
    • Style Transfer ?
    • ONNX ?
      • Emotion detection
    • MS has started integrated deep learning into ML.Net, with TensorFlow














































Shared by joubin on Wed Sep 26 2018

MS Ignite, Azure SQL Database

  • Hyperscale
  • Managed instances
    • Use your own licences
    • Linked Servers
  • Security
    • Enterprise Grade Security
    • Always Encrypted, Encryption in use
    • Encryption in flight
  • Intelligence
    • SQL graph
      • Merge DML
      • View or Derived table in match query
      • Edge Constraints
      • Node tables and Edge tables
    • Intelligent QP
      • Continuously monitoring query performance
      • Spill to disk for queries
      • Spill to TempDb
      • Approximate QP, aggregate queries with data science and analysis and modeling. Not good financial data.
      • Compatibility Level, 150
      • Memory Grant Sizer?
    • Eg Fraud Rings
      • SQL
      • Power BI
  • Reliability
    • Accelerated Database Recovery (ADR)
      • On/Off
    • Parallel PFS updates
    • Resumable online index create, so it won't impact production users
    • Zone Redundancy, greater fault resiliency with replicas
    • Read scale-out support
















Shared by joubin on Tue Sep 25 2018

MS Ignite, AI Oriented Architecture

  • Programming oriented application architecture vs AI oriented app architecture
  • The model is another input into the compute section of an application
  • Enabling Natural User Interfaces (NUI)
  • Ink Intelligence in PowerPoint and Word
  • ML models for learning
    • Handwriting recognition
    • Ink classification
    • Layout analysis
    • Shape recognition
  • Search in Ink
  • Data Ingestion
  • Training
  • Operationalization
  • Project Ink Analysis, Private Preview
  • Why create an AI oriented application
    • Engage employees
    • Engage customers
    • Optimize operations
    • Transform products
  • Azure
    • Pretrained models
    • Frameworks
    • Cora: sales assistant
      • Quite settings
      • Lead scoring
      • Churn prediction
  • Cora, sales bot
    • CRM :-O
  • AI in Patient Flow, staffing and management
  • Health care ML application
  • LOS, length of stay, prediction


































Shared by joubin on Tue Sep 25 2018

MS Ignite, Azure Services

  • Azure Blob Storage
  • Application Insights (AI), APM
    • Azure Monitor
    • Live Steam Metrics
    • Performance
  • Azure Storage Explorer
  • Azure Functions
    • Only pay for the compute time of the function
    • Based on triggers, hooks
    • Can you have multiple functions in a single project that has multiple triggers?
    • Serverless
    • Event driven execution
    • Not used for chatty services
  • Key Vault






Shared by joubin on Tue Sep 25 2018